Hi Liam - Yes extensively but at a different layer. I have done experiments with Codex, Claude Code and Hermes agents both locally and in Azure deployments.
I work at and have been thinking at a different layer as the operations, accounts and project lead rather than dev lead in my company. Without having seen the demo you describe I dont know if this is similarly useful for you but this is the way I have been working. I have noticed it pick up a lot of problems and solve them, make suggestions we hadnt considered etc but you have to read the thinking to see this when you set up an unsupervised plan and work that way as it is nowhere in the outputs. In my opinion Cloud/ Terminal use is not everything because there are tool calling limits and it uses much more tokens than your Pro plans locally. DDSN are moving towards virtual machine farms on our own hardware in a datacenter and a combination of terminal or computer use depending on the application. This means we can use cloud CPU and load balancing but not have the cost of Azure/AWS containers.
We start by training our agent on the process you want, guardrails and company ruleset/ SOP. i.e. train the core agent (e.g. Codex, Claude or Hermes) no matter what the task that your company process is based on design thinking, to keep version control, always be factual and make a plan using subagents and loops rather than work chat by chat - the Soul file. Stop and seek authority the first time you need touch a new system but to operate independently after that. Stop at stage gates before you move on - mostly to make sure we are not wasting tokens going down the wrong track.
We then set up a knowledge base with folders for each software, each client and us with structured and unstructured information that passes the data classification standard - i.e. absolutely no Private information.
Our current agent process is
Expect a short input assets list (5-10 assets max) - if you dont get it ask.
Start by relentlessly questioning me on the process at hand and while that is happening the human user adds additional information that falls out as useful contributions to the inputs list or I might ask it to benchmark the process with research subagents .
Create a plan including using subagents for the work from any inputs - AI drafts / human refines
Confirm the scope - show me the scope a piece at a time in blocks of 5 units so we can refine and lock the scope
Consolidate the user persona - AI drafts / human in the loop confirms
Lets identify all the user journeys - AI drafts / human refines
lets cowork with mermaid diagrams on the information architecture - AI drafts and loops to align with user journeys / human questioning refines
Build the prototype with all our company prototoype style rules - AI drafts and loops to align with user journeys / human questioning refines
AI - task test the prototype walking in the shoes of each of the user journeys and iterate your deficiencies before human review
Produce the specifications of any features we dont currently have in our accelerator used in the prototype
show me the spec one function at a time so we can refine and lock the specification, - AI drafts / human in the loop confirms
ship to asana task list.
Case study 1: Documentation
I used this approach to create our documentation repo for users of Kentico (as opposed to the Kentico one for developers and implementers).
Inputs - I gave it in the knowledge base our safe codebase - i.e. we have split our code into categories we are and are not willing to share with AI (obscurity is still a form of security) and those we are ok with. I also gave it access to our accelerator GUI , the framework documentation and existing K13 user training documentation. The access to our accelerator had a limited role user equivalent to a juinior author and trained it to create our knowledge hub articles.
We first worked together and confirmed the persona of the end users (we have a problem with really poorly skilled staff in our customers and highly skilled IT and we need to serve them both), made & refined the information architecture for the repo, made outlines of each new page in a plan with dotpoint steps I reviewed a sample of these but not all and I approved ahead of letting it go as human in the loop mostly not to waste credits. I let it do a trial of 5 pages with its own QC subagents it actually worked on these 5 for about 2.5 hours working through login issues, tool use issues, gaps in training and mostly wait times - the AI expected a faster experience and reported false failures. After this I gave it additional criticism and improved the page layout , then I let it go for the whole documentation portal over a weekend when I didnt need my computer as the tool calling is a CPU hog. it took 18 hours do do the first pass at the documentation repo. I tried putting it in the cloud with Hermes but it had tool call limits which I hit within 20 min, so Azure container application of this type of activity was not viable. I am pretty proud of this because we got 3 months worth of work done to a higher quality standard in a weekend.
It takes the screenshots, highlights the features of interest with a red square, optimises the image to 800 px width, renames the image, creates the page html which it saves as a backup and authors the page, saves without publishing and saves a record of the completed work at the end for use in future sessions.
At the end I made this a Kentico documentation skill so when someone creates a new feature or we want to extend on Kentico documentation for frontend users who need different infroamtion than developers we give it a prompt with the name of the feature, the page the test content & the tests from QC for this function, if it has a workflow a dotpoint of the workflow and off it goes to create the page. We are also using this after every montlhy Kentico refresh to update the GUI screenshots as well meaning our documentation is kept up to date and it has solved for us the challenge of keeping our client documentation up to date with constant changes to the interfaces.
Case study 2: Content Migration K13 to XbyK
In a recent migration where we were upgrading a client that another Kentico partner built on K13 to XbyK - so the widgets and approach were not compatible with our accelerator and a regular use of the migration tools would not have gotten the job done. I worked with an agent to do the transfer. I used a control experiement for the migration - local computer with Codex with its own credentials (aka not mine). An azure implementation failed because of container tool calling limits imposed by Azure which is why we settled on a computer for the work.
We took the Kentico documentation skill and turned it into a general content skill. Same agent does both documentation now and Kentico authoring.
I created a .env locally to store access credentials for the XbyK development environment user and trained codex to be a junior author - save but not publish. Gave it access to docs.kentico and our K13 documentation. We followed the process, came up with a conversion strategy and gap analysis. I approved a generic page layout and image format. I co-worked through how to use the content hub and channel pages with trials to make sure it knew what it was doing . But I also trained it, how to rename the images, how to run them through AIRA and check the alt text AIRA created and re author with improved brand intent, ensure taxonomy tagging success and resave. We also trained it on what the information architecture rules were for the content hub so it organised images as it went. Once images were transferred thenon pages how to use the content tab to improve the alt text, metatitle and metadescription and widgets - what they were for and what all the fields were for. We also gave it guidelines on how to improve the html (remove unnecessary tags and dividers) and gave it a AA accessibility target.
Then we pointed it to the live site and said transfer all that you can see within your Kentico role which was save but not publish. It did a great job and then our team went through and refined each page. It was more efficient than any migration we have done before - but you do sacrifice some of the metadata.
In the project we also used the agent to do things systematically - for example if a classname on a widget changed the AI was tasked to find it on pages and make a plan to rectify do one test and then execute once a human verified.
Like the last case study we made a Kentico migration skill that is reusable now for the next project. So all we have to work through will be the software improvements on the next go round. Would I use this approach again - yes if I am onboarding a client with a large amount of highly structured content.
Case study 3: QC Testing
If I can solve this it will remove the biggest bottleneck to our use of AI. The dev team keep shipping features with AI assisted coding and Kentico keeps shipping monthly refreshes, and weekly hot fixes and our testing team cant keep up. This has layers - test plan creation from project scope, setting the baselines, running the tests, producing test outcome reportss and shipping bugs to the dev queue.
In this case we have Codex and Hermes (we have tried both), Playwright, Chrome browser use, Open Libre and a Postgres database all working together.
This process we start with our old school test plan document and we extended it to be complete using the same process I listed. Gap analysis from last accelerator QC test plan with new features from project plan and the project contract scope. A human tester reviews the planned tests and ensures the test content is created.
It created a base set of screenshots from the existing production site and functional test content. The AI then created playwright scripts for each of the tests in the library. We create a subset from the library for each project based on scope.
Then there is a test run where it runs the test plan documents the outcome in the document so we can provide the output to the client and produces the Asana task list.
This is in the trial stages (mostly because it either runs out of tokens on a codex plan, or used $300 in tokens in one day with Hermes and ran out of tool calling) If I can finalise the operations for this agent it will become part of our service delivery.
