Working With AI, Not Under It: Trust and Accountability in Generative AI

An AI system that is right 88% of the time is the most dangerous kind to work with. It is wrong often enough to hurt you, and right often enough that you stop checking.

This month our paper Beyond Automation: Generative AI's Influence on Human Decision-Making, Trust, and Accountability in Intelligent Cybersecurity Systems was published in the International Journal of Advanced Research, an HEC-recognised Y category journal, pages 2133-2152. I am one of seven co-authors. The lead author is Abdul Mussawar of the University of Peshawar, and my part, as the author contribution statement records, was supervision, resources and critical revision of the manuscript.

I build AI systems for a living, so I read the paper the way I read any architecture document: where does this break when real people use it under real pressure? This post covers what the paper argues, and how the same ideas already show up in the systems I ship.

What the paper argues

Classic automation follows rules. A firewall rule blocks a port; it does not explain itself. Generative AI is different. It summarises alerts, explains threats, drafts incident responses and ranks vulnerabilities. It produces reasons, and reasons change how a person reads the evidence in front of them.

That is the opportunity: less cognitive load, faster analysis, better use of scarce security talent. It is also the risk. The paper names five problems that come with it:

• Automation bias: accepting the recommendation without checking it.
• Inappropriate trust: relying on the system more, or less, than its real reliability deserves.
• Hallucinated information: confident, fluent output that is simply not true.
• Insufficient transparency: no way to see what the recommendation was based on.
• Unclear accountability: when something goes wrong, nobody owns the decision.

The framework connects the model's capabilities to how humans interpret them, how they verify them, the quality of the final decision and the governance around it. Five factors shape that collaboration: system reliability, professional expertise, workload, access to evidence and governance maturity.

The seven propositions, in plain words

The paper is conceptual: these are expectations to test, not findings from an experiment. Each comes with a suggested way to test it.

Where the paper meets production

None of my products is a security operations centre. But every AI system I have shipped puts a model's output in front of a person who has to act on it, and the same failure modes show up. Here is how I design around them.

1. Trust has to be calibrated in both directions

Most discussions of AI trust worry about people believing the model too much. I learned the opposite lesson first. On my CCTV anomaly detection system, an early phase had a 23% false-positive rate, and the security team started ignoring alerts. Under-trust is a failure too: the real alarm gets lost in the noise. Temporal debouncing and a confidence floor took false positives down to 4%, and the team went from ignoring the system to relying on it. That is P2 in practice: trust tracks reliability, so you fix reliability first.

2. Put a number on uncertainty, and a person behind the threshold

In QuickComm, the voice AI I architected for hotels, an LLM classifies each radio request at about 88% precision. Every output carries a mandatory confidence field. Above the threshold, the task is routed automatically. Below it, the event goes to a human dispatcher. The paper calls this a human approval threshold. In practice it is one line of configuration that decides where a machine may act alone and where a person must decide.

3. Explanations should point at evidence, not just sound convincing

The paper makes a point I agree with strongly: an explanation is only useful if it helps you check the decision, not if it just makes the output look persuasive. That is why my CCTV alerts arrive with a cropped snapshot of what triggered them, and why the research assistant I built, PaperIntel, maps every claim back to a source passage. A reviewer should be able to verify in seconds, not trust in seconds.

4. Give the model a narrow job

The governance model in the paper includes least-privilege permissions. In QuickComm, the LLM never takes an action. It fills a strict JSON schema against a fixed list of intents, the schema is validated at the boundary, and only then does the system route the task. The model proposes; the system validates; a person or a rule decides. A hallucination becomes a rejected payload, not a wrong action.

5. Every decision needs an owner and a record

QuickComm keeps an audit trail of every request, and anything not acknowledged within its SLA escalates from staff to supervisor to head of operations. Nobody can say the system dropped it. That is P6: clear ownership and traceable records are what let a team learn from what went wrong instead of arguing about it.

6. Workload is a safety issue

P7 is the proposition I would most like to see tested in real teams. When people are overloaded, verification is the first thing to go. You cannot fix that with a policy that says check everything. You fix it by sending people fewer, better items to review, which is why I treat false positives as the first problem to solve, not the last.

How I work with AI

The short version of the paper, and of how I build: AI should strengthen human judgment, not replace it. If you are bringing generative AI into a security or operations workflow, this is the checklist I would start with:

• Measure decision quality, not just speed. A fast wrong answer creates more work than it saves.
• Decide before launch which actions the model may take alone and which need a person.
• Show the evidence with every recommendation, so checking is quick.
• Keep the model's permissions as narrow as the job allows.
• Log the input, the output, the confidence and who approved it.
• Give reviewers the authority, and the time, to say no to the model.
• Test on your own data and your own workload, not on a vendor demo.

Why research matters to a builder

Research gives names to problems builders meet every day, and builders supply the messy cases that research needs. Working on this paper with researchers from across Pakistan reminded me how much the two sides need each other. The propositions are now on the record and testable, and I would like to see them measured in real security teams.

If you are putting generative AI into a workflow where wrong answers are expensive, my AI consulting service starts with exactly these questions: where the model may act, where a person must decide, and how you will know it is working.

Code, architecture patterns, and recommendations in this article come from real projects but are shared as-is, without warranty. Validate them against your own requirements before production use. See the Terms of Use.