Post-Delivery Monitoring:<\/p>\n
Set up observability for all live AI systems: workflow error rates, LLM API latency and cost, token consumption per task, retrieval hit rates, and end-to-end task completion rates
\nBuild centralized error logging pipelines,all workflow failures written to Supabase with full context (input, state, error, timestamp), with Teams alerts to the ops team
\nMonitor confidence score distributions on live classification pipelines to detect and investigate distribution shift that may indicate model or data drift
\nUse LLM observability tools (LangSmith,Helicone, orLangfuse) to trace agent runs, inspect intermediate steps, andidentifyfailure patterns in production
\nRespond to client-reported issues,diagnose root causes from workflow logs and agent traces, deploy fixes within agreed SLAs, and communicate clearly with clients throughout
\nProduce monthly performance reports per client: tasks processed, errors caught, confidence distributions, uptime, cost per task, and recommended optimizations
\nProactively identify degradation in model or retrieval quality before clients notice, review evals regularly and propose retraining or prompt updates as needed
\nTool & Vendor Management:<\/p>\n
Manage the engineering stack
\nTrack LLM provider updates, model releases, deprecations, pricing changes, context window expansions, and new capabilities and assess impact on live systems proactively
\nEvaluate emerging agentic frameworks, vector store options, and automation platforms as they mature, recommend adoption with clear rationale and migration plans
\nManage API usage budgets across all LLM and infrastructure vendors: monitor spend, flag anomalies, optimize model selection and caching strategies to control costs
\nMaintain a secure secrets and credentials management system across all environments: API keys, service accounts, OAuth tokens, and database credentials
\nLiaise directly with vendor support and developer relations teams for integration issues,early accessto new features, and technical escalations
\nMaintain an internal tool registry for every tool in the stack, document its purpose, owner, cost, alternative options, and replacement plan in the event of deprecation or failure
\nRequirements
\nTechnical Skills:<\/p>\n
Degree or certification in Computer Science, Software Engineering, Information Technology, or a related field
\n3+ years of software engineering experience, with at least 2 years building production AI or LLM-powered systems
\nStrong proficiency in Python and\/or JavaScript (Node.js) async programming, API integration, data transformation, and working with AI SDKs
\nExperience building with LLMs: prompting, tool use \/ function calling, structured outputs, and streaming responses
\nHands-on experience with LangChain or LlamaIndex, chains, agents, retrievers, memory, and tool integrations
\nHands-on experience with LangGraph, stateful agent design, node\/edge graphs, conditional routing, and multi-agent orchestration
\nWorking experience with vector databases: Pinecone, Weaviate, pgvector, Chroma, or FAISS
\nExperience with REST APIs, webhooks, and system integrations, authentication, retry logic, and rate limit handling
\nFamiliarity with automation platforms: n8n, Make, or equivalent
\nAI \/ ML Knowledge:<\/p>\n
Solid understanding of RAG architectures ,chunking strategies, embedding models, retrieval methods, and re-ranking
\nWorking knowledge of embeddings, semantic search, and vector similarity including trade-offs between embedding models
\nUnderstanding of LLM limitations: hallucinations, context window constraints, prompt sensitivity, and output non-determinism
\nExperience evaluating and improving AI system performance, offline evals, production monitoring, and iterative optimization
\nExposure to open-source LLMs (Llama 3, Mistral, Qwen) and self-hosting inference (Ollama,vLLM, or TGI)
\nInfrastructure & DevOps:<\/p>\n
Experience deploying on cloud platforms: AWS, GCP, or Azure
\nWorking knowledge of Docker and CI\/CD pipelines for AI workload deployment
\nFamiliarity with monitoring and observability tools such as, LangSmith, Langfuse, Helicone, or Arize Phoenix
\nExperience with Supabaseor PostgreSQL for application data and pgvector for embedding storage
\nNice to Have:<\/p>\n
Experience with agent frameworks: AutoGen, CrewAI, or custom multi-agent orchestration patterns
\nExperience with workflow orchestration tools: Temporal, Celery, or BullMQ for long running async processes
\nFamiliarity with model fine-tuning: LoRA\/QLoRA, dataset curation, and evaluation of fine-tuned outputs
\nExperience with hybrid search: combining dense vector retrieval with BM25 \/ sparse retrieval and cross-encoder re-ranking
\nExperience withTermii\/Twilio WhatsApp Business API for conversational agent delivery
\nBackground in building SaaS products, internal tools, or multi-tenant applications with data isolation
\nExperience working in a startup or agency environment with comfortable context-switching across multiple client projects
\nUnderstanding of security, rate limiting, secrets management, and production reliability practices
\nContributions to open-source AI agent frameworks, LLM tooling, or automation projects<\/p>\n","protected":false},"author":22,"featured_media":0,"template":"","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"_promoted":"","_job_location":"Lagos","_application":"careers@jedayahAI.com","_company_name":"Jedayah AI","_company_website":"http:\/\/.","_company_tagline":"","_company_twitter":"","_company_video":"","_filled":0,"_featured":0,"_remote_position":0,"_job_salary_currency":"","_job_salary_unit":""},"job-types":[3],"class_list":{"0":"post-20906","1":"job_listing","2":"type-job_listing","3":"status-publish","4":"hentry","5":"job_listing_type-full-time","7":"job-type-full-time"},"_links":{"self":[{"href":"https:\/\/www.jobsinnaija.com\/index.php?rest_route=\/wp\/v2\/job-listings\/20906","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.jobsinnaija.com\/index.php?rest_route=\/wp\/v2\/job-listings"}],"about":[{"href":"https:\/\/www.jobsinnaija.com\/index.php?rest_route=\/wp\/v2\/types\/job_listing"}],"author":[{"embeddable":true,"href":"https:\/\/www.jobsinnaija.com\/index.php?rest_route=\/wp\/v2\/users\/22"}],"wp:attachment":[{"href":"https:\/\/www.jobsinnaija.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=20906"}],"wp:term":[{"taxonomy":"job_listing_type","embeddable":true,"href":"https:\/\/www.jobsinnaija.com\/index.php?rest_route=%2Fwp%2Fv2%2Fjob-types&post=20906"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}