ParakhAI

AI systems are increasingly shaping critical decisions across public health, finance, governance, and climate. These high-risk AI models influence access to benefits, services, and opportunities for individuals and communities. As highlighted in Stanford’s AI Index Report 2025, reported AI-related incidents surged by 56.4% in 2024, reflecting growing concerns around safety and reliability.

While regulatory frameworks are evolving, current mechanisms often rely on self-assessment and lack robust multi-stakeholder accountability. These challenges arise due to biases in training data, technical limitations, and societal or individual prejudices embedded within AI systems. Such risks can undermine public trust and limit the effective integration of AI in governance and decision-making.

There is a need to operationalise safe, trustworthy, and responsible AI systems through participatory evaluation approaches that move from high-level principles to applied, context-sensitive solutions, particularly reflecting Global South realities.

CivicDataLab has co-created ParakhAI, an open-source, participatory tool for AI evaluation. The tool is supported under the Ministry of Electronics and Information Technology’s IndiaAI Mission, under the Safe and Trusted AI pillar.

ParakhAI is designed to identify bias, hallucination, and informational integrity issues in AI models to support responsible and trustworthy deployment. It incorporates participation from developers, technical experts, domain specialists, and cultural experts across the AI lifecycle, moving beyond self-regulation.

The tool combines automated evaluation modules with human-in-the-loop assessments and community-led approaches, enabling comprehensive evaluation and standardised report generation. It includes a web-based evaluation platform, prompt dataset curation, a guidebook for evaluation practices, and pilot applications across sectors such as public health, agriculture, language, climate, and public services.

The platform is built as a scalable, API-first system that supports multiple AI models and providers, enabling flexible and consistent evaluation across use cases. It allows asynchronous execution of evaluations, secure access through role-based permissions, and robust tracking of performance metrics which ensures reliability, transparency, and adaptability as the ecosystem evolves.

Alongside this, CivicDataLab is supporting capacity building through workshops and open resources to enable wider adoption of participatory and transparent AI evaluation practices.