Digital Alchemy Lab

Digital Alchemy Lab

Share this post

Digital Alchemy Lab
Digital Alchemy Lab
Teaching AI to Spot Harmful Content: A Journey into Unsupervised Learning
Copy link
Facebook
Email
Notes
More
AI Chronicles

Teaching AI to Spot Harmful Content: A Journey into Unsupervised Learning

Building in Public: Part 1 - Setting Up Our Training Pipeline for Content Detection

Devon Artis's avatar
Devon Artis
Nov 01, 2024
∙ Paid
5

Share this post

Digital Alchemy Lab
Digital Alchemy Lab
Teaching AI to Spot Harmful Content: A Journey into Unsupervised Learning
Copy link
Facebook
Email
Notes
More
Share

Hey there, curious minds! 👋

Welcome to the first installment of my "Building in Public" series. While everyone's chatting about ChatGPT writing poems and DALL-E creating art, I'm taking you behind the scenes of building something different - an AI system that helps identify harmful content. I'm documenting every step, every challenge, and every "aha!" moment as part of the Bias Bounty 2 challenge by Humane Intelligence x Revontulet.

Why Understanding AI Bias Matters for Everyone 🌍

Let's have a real talk about AI bias and explainability. It's not just a technical problem - it's a societal challenge that affects all of us. Think about it:

  • Your loan application might be processed by AI

  • Job applications often go through AI screening

  • Social media content is moderated by AI

  • Healthcare diagnoses are increasingly aided by AI

Every single one of these systems can have biases, and these biases affect real people's lives. But here's the thing: you don't need to be a programmer to be part of the solution.

Finding Your Place in the AI Revolution 🔍

There are many ways to contribute to better, more ethical AI:

  1. Business Leaders: Understanding AI bias helps make better decisions about implementing AI systems

  2. Product Managers: Knowing about AI explainability helps design better user experiences

  3. Content Creators: Understanding how AI moderates content helps create better, inclusive content

  4. Policy Makers: Grasping AI bias helps create better regulations

  5. Users: Being aware of AI bias helps us better navigate and question the systems we interact with daily.

What Does "Building in Public" Mean? 🤔

Imagine if a chef not only served you the final dish but also:

  • Showed you their grocery shopping list

  • Let you watch them chop every vegetable

  • Explained why they chose certain ingredients

  • Shared when they burned the first batch

That's what we're doing here with AI! I'm sharing:

  • Every line of code (even the messy first drafts)

  • The thinking behind each decision

  • The dead ends and failed attempts

  • The successful breakthroughs

Digital Alchemy Lab is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

The Technical Journey Begins 🚀

Now, let's roll up our sleeves and look at how we're actually building this. We're using Python and focusing on Apple Silicon Macs - but don't worry if you have different hardware, I'll cover alternatives in future posts.

Our Toolkit 🛠️

First, let's look at our training script (train.py):

python
"""
Training script for hate content detection model
"""
import tensorflow as tf
from utils import ImageProcessor, SimpleMetalKMeans

def train():
    """Main training function"""
    print("Starting hate content detection model training...")
    
    # Print TensorFlow device info
    physical_devices = tf.config.list_physical_devices('GPU')
    print("Available devices:", physical_devices)

Think of this as setting up our kitchen before cooking. We're:

  1. Importing our tools (TensorFlow and custom utilities)

  2. Checking if we have our high-speed processor (GPU) available

The Image Processor: Our Digital Photo Assistant 📸

python
# Initialize processor
processor = ImageProcessor()

# Load and process training data
print("\nLoading training data...")
train_folder = './Training_data'
train_images, train_ids = processor.load_images(train_folder)

What's happening here? Imagine you're preparing photos for a digital album:

  1. The ImageProcessor is like your helpful assistant

  2. It takes each image and:

    • Resizes it to 64x64 pixels (like making sure all your photos are the same size)

    • Converts it to the right color format

    • Uses multiple workers (like having several assistants helping at once)

Keep reading with a 7-day free trial

Subscribe to Digital Alchemy Lab to keep reading this post and get 7 days of free access to the full post archives.

Already a paid subscriber? Sign in
© 2025 Devon Artis
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture

Share

Copy link
Facebook
Email
Notes
More