Fictitious Precedents: The Catastrophic Cost of Unverified AI in the Courtroom

Introduction

Generative artificial intelligence (AI) has held great promise for a commercial legal practice, promising unprecedented efficiency. However, with the rapidly growing number of technologies, there are more and more risks of structural damage. The Supreme Court of India has recently considered the issue of AI generated "hallucinations" getting recorded in court. The Bench of Justices P.S. Narasimha and Alok Aradhe in Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd. gave a severe reprimand to the legal fraternity, setting aside important insolvency orders, for being based on “fictitious” precedents, as stated in the LiveLaw Case Report of Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd.

The Case Background: How a fictional precedent tainted two Tribunals

The controversy started with a typical corporate insolvency proceeding under Section 7 of the Insolvency and Bankruptcy Code (IBC), 2016. Jammu & Kashmir Bank Ltd. (JKBK) had filed corporate insolvency resolution process (CIRP) against Essel Infraprojects Ltd. (EIL) for its role as a corporate guarantor in providing loan facilities to its associate, Pan India Utilities Distribution Company Ltd;

The Section 7 application was filed before the National Company Law Tribunal (NCLT) and later challenged before the National Company Law Appellate Tribunal (NCLAT) which ruled in favor of the NCLT, following its reasoning in the Section 7 application[2]. But when the suspended director of the corporate debtor challenged this before the Supreme Court, it was discovered that there was a glaring procedural contamination – six case precedents cited by the NCLT and NCLAT out of which the courts found that there was no real case or case citations that were stuffed with fabricated paragraphs generated by AI.

Jammu and Kashmir Bank had in its affidavit stated that its lawyers had never used these six falsehoods to support its case. As reported in The Wire's article on Legal AI accountability, these precedents were fabricated by the tribunal itself, based on its unchecked AI-powered research.

The Supreme Court's Decisions: "No Decision in the Eyes of the Law"

Supreme Court expressed grave concern about loss of the integrity of the Judiciary. The Court, in the order setting aside both the NCLT and NCLAT orders, and remanding the Section 7 application to the NCLT for fresh consideration, laid down a few important principles of the modern commercial world:

• In the official judgment text of Indian Kanoon, the Court stated that the tribunals and courts ought to have a ‘zero tolerance' stance towards citing or using or relying on AI-generated precedents without a proper human input verification process. The Court in the official judgment text of Indian Kanoon held that tribunals and courts should adopt a ‘zero tolerance' approach towards citing or producing or relying on AI-generated precedents without adequate human input verification process.

• The Adjudicatory Nullity Rule: If a judge or quasi judge makes a decision based on false or hallucinated legal material, it is no decision at all, the Bench declared. The judgment is tainted, it is not sustainable and therefore must be scrapped if it is based on a "iota" of "hallucinated precedent".

• The Court used the analogy of the "Bhopal Gas", emphasizing that unverified technology in the Judiciary is insidious, invisible and catastrophic when it comes to notice. The Court drew a clear comparison and cited that the introduction of fake precedents in judicial decisions is like the release of the methyl isocyanate in the province of law and justice, insidious, invisible and catastrophic by the time it comes to notice.

• While recognizing that lawyers are forced to embrace new technologies that utilize AI in order to cope with the mounting cases that are being handled, the Court maintained that the human involvement in the adjudication process must be absolute and unconditional at each and every stage.

  
  

Main Takeaways for Corporate Lawyers & In-House Counsels

The following are some of the main takeaways corporate lawyers & in-house counsels might have from this article:

In the case of Pooja Ramesh the Supreme Court has provided a paradigm shift in conducting legal research and technology for commercial litigation within an in-house legal department:

• The country's highest court has officially deemed the use of AI-generated but fake case precedent for citation to be professional misconduct. The Supreme Court has instructed the Bar Council of India (BCI) to form a committee to establish hard guidelines on any practitioner before courts presenting unverified written authorities generated by AI tools which are in violation of the rules of conduct.

• For document review, AI synthesis or document drafting, it is suggested to use AI tools efficiently, but relying on AI without verifying it is a big risk. Before pleading, corporate legal teams need to adopt a strict internal verification scheme and ensure that the appropriate internal records (including official records of the original judgments) are pulled.

• In the present case, the tribunals themselves conducted the faulty research, hence making adherence to the same duty of care on the adjudicators as well. In the current case, it is the duty of care of the tribunals as well, since the tribunals themselves conducted the flawed research. During court hearings, commercial courts will likely begin requesting official or verified physical and/or digital citations for any arcane or more complex legal proposition.

The OpenAI Paradox: Shifting Liability and the Battle Over the Legal Workflow

This dichotomy of efficiency vs accountability has grown stronger as OpenAI has taken a stance as a legal vertical conqueror in addition to its role as a basic infrastructure provider. Instead of using only startups to create legal-specific wrappers, OpenAI went so far as to recruit Jason Boehmig, the previous CEO of contract management platform Ironclad, to head a dedicated product team for legal. It's a clear attempt at land grab in the legal workflow itself, and something that suggests that tech giants aren't going to back away from traditional legal workflows, instead trying to introduce automation into high-context, complex processes.

The promise of hyper-efficient case-law analysis, lower billable hours and fast generation of contracts draws the value proposition of law firms and corporate legal departments. But, like in the example of Pooja Ramesh Singh, the Supreme Court of India has explained in the case, this race to automation adds to the complexities of liabilities.

This disruption is starkly exposed in its dramatic new product liability case, Nippon Life Insurance Company of America v. OpenAI from early 2026, which brought together multiple parties in the product liability and unauthorized practice of law realm.

If so, a former claimant avoided using a lawyer, typing her attorney's correspondence into ChatGPT and utilizing the software to write dozens of post-settlement court papers, some of which included bogus case citations. Nippon Life filed a lawsuit against OpenAI for $10.3 million that claims the developer knowingly allowed an unlicensed advocacy system to be created by its consumer-facing AI system, without an adequate “refusal architecture” to prevent this.

The legal tech industry thus finds itself in a conundrum: This poses an existential problem for legal tech: OpenAI and other players, like Anthropic, are seeking to sell "intelligence as a utility" directly to law firms, but their products that are easy-to-use by consumers are moving beyond the “uncrossable threshold,” which legal technologists describe as the line between the passive, unactive, and harmless collection of legal information and the active, autonomous, and unverified generation of legal advocacy. Once AI reaches this point it becomes a liability engine.

Don't rely on a machine to carry out the verification, lessons from Pooja Ramesh Singh and Nippon Life are the same for corporate attorneys. Any lawyer who takes any of the “direct legal” offerings from OpenAI without 100% validation by a human lawyer is a professional suicide bomber. AI isn't only going to take the place of lawyers, it's going to ruin the careers of those that blindly use AI tools that aren't verified. As Big Tech is seeking to dominate the legal market, legal professionals are just about the only thing that can stand in the way of the encroachment is the simple application of human judgment. AI as an invisible, insidious and catastrophic threat to justice cannot be staved off by any one of the following in isolation: absolute human control over every sentence, citation and submission.

Conclusion

The case of Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd. is one of the most prominent judgments of the digital age of corporate commercial law. It makes a clear distinction between technology as an enabler for efficiency and technology as a replacement for the human mind. The takeaway for corporate litigators is that the use of AI should be done responsibly, and that diligence in checking each and every citation is absolutely essential; especially when it comes to integrity in the application of the law.