Can Telemetry Replace the Exam?

Most IT certification still rests on a snapshot: an exam that asks whether you know something, on one day, under test conditions. But knowing and doing are not the same thing — and in a field where the tools change weekly, a snapshot ages fast.

On this episode, Tim Burnett is joined by Jim Henderson, owner of Validify Consulting and a Seattle-based specialist who helps IT companies build high-stakes certification programmes. With around twenty years in assessment — and a career in IT before that — Jim makes the case for assessing competence in the flow of work.

"IT professionals were very interested in not just showing that they knew something, but showing that they could do something."
— Jim Henderson, Owner, Validify Consulting

What is "assessment in the flow of work" — and why is it interesting?

Jim's interest goes back to his time at Novell (2005–2010), where he ran the "practicum" — a performance-based, remotely delivered exam at a time when remote testing was still novel and genuinely hard to build. The principle stuck: measure what people actually do, not just what they can recall.

Cloud software and ubiquitous connectivity now make that far more feasible. Every SaaS platform quietly records how users behave — data normally used by product teams to decide what to build next. Jim's insight is that the same data could tell you whether someone is performing a task competently. It's the natural counterpart to Josh Bersin's "learning in the flow of work" — applied to assessment.

Dimension	Traditional knowledge exam	Assessment in the flow of work
What's measured	Recall, at a point in time	Real tasks completed in a live system
When	A single high-stakes session	Continuously, over weeks or months
Staying current	Ages as the platform changes	Tests whether you keep up with change
Main integrity risk	Cheating, leaked items, proxies	Scripting or gaming the measured signal

What is telemetry, and how does it become an assessment?

Telemetry is simply the logging that happens whenever you perform a task in a software system — a record of who did it, when, and whether it succeeded. Increasingly it's event-driven data that can be subscribed to, not just a log file sitting on a server.

Crucially, it's atomic. Coming from an identity-and-access-management background, Jim's example is a single event that records that "Jim created this user, at this time, in this part of the directory." The skill is not to watch every click, but to read the end state — inspecting log entries, configuration files and back-end data, then running a grading script of true/false checks to confirm the task was genuinely done.

"You know that the user was a real user, and not someone that Jim created just so he could tick the box and say he did the task."
— Jim Henderson, Owner, Validify Consulting

In other words, if a created user then logs in and resets their own password, the evidence chain proves the work was real — not staged to satisfy a checklist.

What can assessment learn from video games?

The spark was a game Jim plays with his son. Its progression system tells you at a glance how competent another player is — essential when you're matched with strangers all over the world.

The design detail is striking: an 11-tier system, with 15–20 objectives per rank, running from basic tasks at the bottom to the hardest activities in the game at the top. To Jim, it "looked an awful lot like building an exam blueprint." And best of all, it resets each season.

"You had to effectively recertify as you continue to play the game."
— Jim Henderson, Owner, Validify Consulting

Key takeaway: the seasonal reset designs currency into the credential — a rank reflects skill now, not something earned years ago.

Can you build SaaS platforms to be "exam-ready"?

Tim asks whether SaaS tools will start shipping with assessment-ready telemetry baked in. Jim's view is that plenty of usable telemetry already exists, but privacy rules and internal data-separation requirements can stop even employees getting at it.

The bigger obstacle is pace. Jim has watched a platform feature go end-of-support two months out while a form-based exam covering it was mid-build. His answer is to treat assessment telemetry as a first-class part of the product — with unit tests that flag "this data is also used to assess people's skills; don't break it."

There's also a data-retention trap: telemetry volumes are so large that retention windows are often short — perhaps 30 days — while renewals run quarterly or yearly. The evidence can vanish before it's needed. Jim points to ISO- and ITIL-style process certifications and chain-of-custody thinking as the way to keep an auditable record.

Is telemetry more secure than proctoring — and can it be gamed?

"I'm going to steal an answer from the psychometrics field and say: it depends."
— Jim Henderson, Owner, Validify Consulting

On the plus side, using live production systems has security baked in. Handing your admin credentials to someone else isn't a harmless workaround — in regulated sectors like finance or healthcare it can breach your terms of employment, so identity is protected by real-world stakes.

The "it depends" is about design. A crude, atomic objective — "create five users" — can be scripted once and forgotten, which says little about sustained competence. The fix is to spread the evidence over time: require, say, five users created over six months, with a minimum gap of a month between each.

"We're not just measuring their skill in 'create user' — we're also measuring their skill in being able to adapt to changes in the environment."
— Jim Henderson, Owner, Validify Consulting

Key takeaway: integrity comes from how you design the objectives and how long you consider them valid — not from any single control. Telemetry and proctoring aren't rivals so much as ingredients.

Could certification become tiered and stackable, like levelling up?

Tim wonders whether this shifts us from pass/fail towards something richer. Jim thinks so: credentials can be built as tiers, or as stackable and parallel micro-credentials mapped to different job roles, with objectives that nest — create 5, 10, then 15 users as levels one, two and three.

To test the idea, Jim has built a working prototype with an API ingestion endpoint and a data simulation, so he can watch the tiers fill in as data flows through.

"If I can see it doing it, that helps me actually think about it."
— Jim Henderson, Owner, Validify Consulting

Where could this go — and what to do next

Jim isn't working in isolation — Microsoft's Liberty Munson has been exploring similar ground. Nobody can say exactly what IT certification looks like in two to five years, but the direction of travel is clear: towards richer, more authentic, more current evidence of what people can actually do.

If you want to explore the idea:

• Start with validity, not data. Define what real competence looks like in your platform before deciding what to capture.
• Design objectives across time. Spread evidence over weeks or months to resist gaming and prove durable skill.
• Treat assessment telemetry as a product feature. Protect it with unit tests and a defined retention and chain-of-custody policy.
• Watch Jim's prototype demo (linked in the show notes) to see the tiered model in action.

Referenced in this episode: Validify Consulting at the ITCC 2026 Vendor Showcase — plus Jim's prototype demo (add the video link here for the show notes).

Connect with Jim Henderson on LinkedIn: linkedin.com/in/jhenderson

Connect with Tim Burnett on LinkedIn: linkedin.com/in/tburnett

Subscribe to the Test Community Network: testcommunity.network