YouTube’s “Trending” page is broken for researchers. It surfaces videos with the most views—many of which went viral days or weeks ago. What if you only want what’s new?
This is the problem I solved for my content intelligence workflow. Here’s the technical journey.
When I tried to get recent YouTube videos programmatically, I ran into a wall. Standard scraping tools give you popularity-ranked results, not time-ranked results.
My first attempt: use yt-dlp with a date filter. The tool supports --dateafter YYYYMMDD, which sounds perfect. But here’s what actually happened—YouTube’s search results don’t expose upload dates through that interface. Every video came back with upload_date = NA.
Second attempt: hit the YouTube Data API. It requires an API key, has quota limits, and the free tier is restrictive for any real monitoring use case.
Third attempt: render the page with a headless browser and extract what I needed from the DOM. This is where I found success—but it came with its own complications.
YouTube is a single-page application. The video list, the filters, the timestamps—all of it is rendered client-side by JavaScript. When you fetch the raw HTML, you get a near-empty shell. The content appears after the JavaScript executes.
There’s another layer: YouTube uses Web Components with Shadow DOM. The filter tabs aren’t accessible through normal DOM queries. You have to traverse shadow trees to find and click them.
The timestamps themselves are interesting. YouTube renders them as relative strings—“3 days ago”, “2 weeks ago”—in the user’s locale. If you’re scraping from a non-English context, these strings are localized and nearly impossible to parse reliably.
I used Playwright to launch a headless Chromium instance with explicit locale settings:
1 | from playwright.sync_api import sync_playwright |
Setting en-US locale does two things: it forces YouTube to render timestamps in English (“3 days ago”) and it makes the page behavior consistent regardless of where the scraper runs.
The “Recently uploaded” tab sits inside YouTube’s custom elements. Standard selectors won’t reach it. I wrote a recursive traversal function:
1 | function walk(el, depth) { |
This walks the entire shadow tree, finds the tab element, and clicks it. After the click, YouTube AJAX-refreshes the video list. I wait a few seconds for the new results to render.
Once the “Recently uploaded” results are in the DOM, I extract video titles, IDs, and timestamps using a single JavaScript evaluation:
1 | () => { |
Then in Python, I parse the relative time and filter to 7 days:
1 | import re |
Only videos uploaded within the last 7 days pass through. Older content is discarded automatically.
The result is a lightweight monitoring pipeline: search topics I’m interested in, get only the videos uploaded this week, and push them to a destination of choice. No API keys, no quota limits, no reliance on third-party wrappers.
For content researchers, this means surfacing emerging topics before they hit the mainstream trending list. For product teams, it means tracking how new releases spread in real time.
The core lessons: when a site is fully client-side, render it like a browser. Control your locale. Traverse shadow trees explicitly. And parse relative time at the source—don’t try to reconstruct it from data that was never there.
]]>There’s a moment every developer hits when building AI agents. It usually comes after the third “helpful” hallucination, the fifth off-task tangent, or the first time your agent decided to delete production data because it “seemed efficient.”
That’s when you ask the question that haunts every agent project:
How do you control something that’s smarter than you, but also unpredictable enough to burn everything down?
This is the core dilemma of AI agent development. And it’s the reason the industry keeps talking about “harnesses.”
A harness is the system you build around an agent to keep it useful without letting it cause damage. Think of it like the safety systems in a car — airbags, seatbelts, ABS brakes. You don’t drive without them. But they also change how you drive.
In agent terms, the harness includes guardrails that prevent certain actions, context windows that manage how much history the agent can work with, fallback systems for when things go wrong, and monitoring layers that track what’s actually happening.
The harness sits between the agent and the real world. The agent decides. The harness validates, redirects, and cuts off when necessary.
Here is the fundamental tension: the properties that make agents useful are the same ones that make them dangerous.
An agent that can reason across domains, use tools, and take multi-step actions is extraordinarily powerful. But that flexibility is a double-edged sword. The agent isn’t just going where you want — it’s going where it thinks is best, based on a goal you gave it that may not perfectly match reality.
This is the “helpful but unpredictable” problem.
Early agent demos always look impressive because they’re running in controlled environments. The agent is given a clear goal, a clean context, and room to explore. But production is messy. Users say unexpected things. External data changes. Edge cases surface. And the agent, left unchecked, starts making assumptions that seem reasonable but lead to garbage.
The harness exists because the real world is not a demo.
Every harness involves a tradeoff: the more control you add, the more you limit what the agent can do. The less control, the more risk you take on.
This is the autonomy-control tradeoff, and it’s the central engineering challenge of agent development.
At one extreme, you could hard-code every decision an agent makes. No flexibility. No reasoning. Just rules. That’s safe, but it’s not an agent — it’s a script.
At the other extreme, you give the agent full autonomy. Maximum capability. Maximum risk. Maximum potential for it to do something you didn’t anticipate.
The harness is where you find the balance. You give the agent enough rope to be genuinely useful, but not enough to hang itself.
This isn’t a solved problem. Every team doing serious agent work is trying to figure out where that line is. And the answer keeps changing as models get more capable — what felt safe last year feels risky this year.
The autonomy-control tradeoff is manageable when you’re running one agent. You can watch it. You can understand what it’s doing. When something goes wrong, you can trace it.
But agent systems don’t stay small. They scale. One agent becomes five. Five becomes a system of agents working together, handing tasks off, sharing context, building on each other’s outputs.
At scale, the harness becomes load-bearing. Without one, you don’t just have risk — you have chaos. Agents working at cross-purposes. Task loops that eat resources. Error cascades where one agent’s mistake propagates through the whole system.
The harness at scale is infrastructure. It’s the difference between a team that coordinates and a team that gets in each other’s way.
A well-designed harness does five things:
1. Prevents catastrophic outputs — Filters dangerous or nonsensical results before they reach users. This is the baseline. No agent goes to production without this.
2. Corrects drift — If an agent starts going off-topic or losing focus, the harness redirects it. Think of it like a thermostat: it doesn’t control the system directly, it keeps it within bounds.
3. Manages context — Long conversations degrade agent performance. The harness handles context compression, prioritization, and memory — keeping the agent sharp even after hundreds of turns.
4. Handles failures gracefully — When something goes wrong — a tool fails, an API times out, the agent hits something it can’t handle — the harness catches it. It retries, escalates, or degrades gracefully instead of leaving the user with a broken experience.
5. Gives you observability — You can’t control what you can’t see. The harness tracks agent behavior, logs decisions, and surfaces patterns so you can debug and improve over time.
Here’s what makes harnesses genuinely difficult to build: the right level of control changes over time.
An agent that’s working well on Monday might start drifting on Tuesday if the external context shifts — new products, new user queries, new data. A harness that was calibrated correctly in January might be too loose by March.
The harness can’t be static. It has to adapt. Which means you’re building monitoring into the monitoring system, and then monitoring that.
This is why agent reliability work is so undervalued and so underestimated. It’s not glamorous. It doesn’t produce impressive demos. But it’s what separates agents that work in demos from agents that work in production.
The core dilemma — needing to control agents, but that control threatens to undermine what makes them useful — is not a bug. It’s a fundamental property of working with systems that are genuinely intelligent and genuinely unpredictable.
The question isn’t whether to harness agents. You have to. The question is how to build harnesses that make agents more useful, not less — that constrain them enough to be safe, but leave them enough room to be genuinely smart.
That’s the actual engineering problem. And it’s one that every team building agents is still figuring out.
The good news: the harness exists because the agent is worth protecting. If the dilemma didn’t exist — if agents were easy to control — they’d probably be too simple to be worth controlling.
The harness is a sign that you’re working with something powerful. The hard part is making sure the power stays pointed in the right direction.
Related Articles
]]>This scenario has played out in countless people’s lives over the past year. Anxiety is like a chronic cold—not fatal, but it drains your focus and judgment nonstop. You’ve tried learning Python, watching AI tutorials, and signing up for all kinds of courses, but the more you learn, the more anxious you get—tech updates too fast to keep up with. You’ve also tried the “chill out” mindset, telling yourself “everything will work out”, but every time you see an AI-related news story, the anxiety creeps back in.
What’s really missing here? Why does fighting anxiety harder only make it stronger?
The most common tip: Learn tech, keep up with the times.
It sounds logical. But the problem is, you can never outpace AI’s iteration speed. GPT-4 only came out a year ago, and GPT-5 is already on the way. You just mastered Midjourney, and new image-generation tools drop. Learning tech becomes a bottomless pit; the more you learn, the more you feel behind.
Another common tip: AI is just a tool; it won’t replace humans.
This also makes sense. But look at the roles being automated—data entry, junior translation, basic copywriting—and you’ll see “tools” are quickly swallowing these jobs. In 2023, IBM replaced 7,800 roles with AI; in 2024, an e-commerce platform used AI to write product descriptions, cutting its content team by 60%. Is this still just a “tool”?
Others tell you: Anxiety is useless; just take action.
It sounds positive. But have you noticed this “action” is often blind? You sign up for three AI courses, buy two paid tools, spend two hours a day learning, but your anxiety doesn’t fade. Because your actions have no direction—you’re just masking fear with busyness.
The shared flaw of all this advice: They all try to fix internal anxiety with external actions, but the root of anxiety isn’t there.
In truth, the core of workplace AI anxiety is: a loss of control.
You feel your career fate is being rewritten by an invisible force, and you have no clue what the rules are. You don’t know which skills will become obsolete, which roles will disappear, let alone where to focus your efforts. This loss of control is the real source of your anxiety.
So, the key to beating AI anxiety isn’t learning more tech—it’s rebuilding your sense of control—knowing what to hold onto, what to let go of, and where to go.
Career transition: Most people hesitate to switch jobs not for lack of skills, but because they don’t know the rules of a new industry. Headhunter data shows 80% of workers feel anxious for 3–6 months before switching roles; their anxiety fades only when they enter the new field and build a new sense of control.
Investment: Retail investors panic most when the stock market crashes—they don’t know where the bottom is, whether to cut losses or buy the dip. Professional investors stay calmer because they have clear strategies and stop-loss lines—they know exactly what to do.
Health issues: You’ll stress for months if a checkup shows an abnormal marker and the doctor says “just monitor it”. But anxiety vanishes instantly if the doctor clearly tells you “it’s benign, no impact on daily life”. Uncertainty is anxiety’s amplifier.
Historical case: During the Industrial Revolution, textile workers smashed machines because they thought machines stole their jobs. Later, workers learned to operate machines and became skilled laborers with higher pay. They went from “replaceable workers” to “machine operators” and reclaimed control.
AI field: People who actually use AI to boost efficiency rarely feel anxious. They know what AI can and can’t do, and where their own value lies. They treat AI as a lever, not a threat.
All these cases point to one conclusion: The antidote to anxiety isn’t avoidance or blind action—it’s building a new sense of control.
A survey of 1,000 workers found:
The gap isn’t learning time—it’s having clear scenarios and a sense of control.
Another data point: A recruitment platform’s stats show that in the first half of 2024, hiring for emerging roles like “AI Efficiency Specialist” and “AI Product Manager” surged 300%. Salaries for these roles are 20–40% higher than traditional roles on average. This means: People who use AI as a tool are getting paid more; people replaced by AI are losing value.
Step 1: Inventory your irreplaceable strengths
List your three core job skills. Ask yourself: If AI can do 50% of each, what’s left? That remaining part is your core to protect. It could be industry experience, professional connections, complex decision-making skills, or that “human touch” AI still can’t replicate.
Step 2: Find AI’s limits
Don’t learn AI vaguely. For your job, clarify exactly what AI can and can’t do. For example, AI can generate draft copy but can’t grasp brand tone; AI can analyze data but can’t make business strategies. These limits are where you position yourself.
Step 3: Design your AI collaboration model
Stop thinking “how to not be replaced by AI” and start thinking “how to use AI to work smarter”. Treat AI as a tool and redesign your workflow. Once you boost efficiency by 30%+ with AI, you go from “threatened employee” to “controller”.
Anxiety won’t disappear, but you can turn it into a clear direction for action.
Beginner (3 books)
Advanced (4 books)
Academic (3 books)
Anxiety is a signal, not an enemy. It’s reminding you: It’s time to rebuild your sense of control.
Related Articles
]]>