Automating Feature Class Count and Metadata Export with Python and ArcPy

If you are working with large geospatial datasets in Esri’s ArcGIS, keeping track of the features within each geodatabase is essential. For example, you may need a quick overview of feature class types, their counts, and the shape types across all geodatabases in a directory. In this blog post, I’ll show you how to automate the process of gathering feature class metadata and exporting it to a CSV file using Python and ArcPy.

The solution I’ll demonstrate scans all geodatabases in a specified directory, collects metadata (such as the feature class name, dataset, shape type, and feature count), and exports it to a CSV file. This approach helps you quickly summarize key attributes of all feature classes in your project.

Use Case

You might need this kind of automation when:

• You want to document all feature classes in your geodatabases.

• You need to review the shape types and feature counts of all geospatial data in a project.

• You’re preparing reports or verifying data integrity across large geodatabases.

The Python script below automates this process by walking through a directory of geodatabases and saving the results in a CSV file for later use.

Code:

python
import arcpy
import os
import csv

# Define the folder path containing the geodatabases
folder_path = r'PATH_TO_YOUR_GEODATABASES_FOLDER'  # Update with your geodatabase folder path

# Define the output CSV file path
csv_file = r'PATH_TO_YOUR_OUTPUT_CSV'  # Update with your desired CSV file path

# Define the CSV headers
csv_headers = ['GDB Name', 'Dataset', 'Featureclass Name', 'Shape Type', 'Count']

# Open the CSV file for writing with utf-8 encoding
with open(csv_file, mode='w', newline='', encoding='utf-8') as file:
    writer = csv.writer(file)
    writer.writerow(csv_headers)

    # Walk through the directory tree using os.walk()
    for root, dirs, files in os.walk(folder_path):
        for dir_name in dirs:
            if dir_name.endswith('.gdb'):  # Check if the directory is a geodatabase
                gdb_path = os.path.join(root, dir_name)

                # Set the workspace to the current GDB
                arcpy.env.workspace = gdb_path
                print(f"Processing {dir_name}")

                # List all standalone feature classes (those not in datasets)
                standalone_featureclasses = arcpy.ListFeatureClasses()

                for fc in standalone_featureclasses:
                    fc_path = os.path.join(arcpy.env.workspace, fc)
                    desc = arcpy.Describe(fc_path)

                    # Get the shape type and feature count
                    shape_type = desc.shapeType
                    count = arcpy.GetCount_management(fc_path)[0]

                    # Write to CSV (No dataset for standalone feature classes)
                    writer.writerow([dir_name, 'None', fc, shape_type, count])

                # List all feature datasets in the geodatabase
                datasets = arcpy.ListDatasets('', 'Feature')

                # If datasets exist, iterate through them
                if datasets:
                    for dataset in datasets:
                        # List feature classes in each dataset
                        dataset_featureclasses = arcpy.ListFeatureClasses(feature_dataset=dataset)

                        for fc in dataset_featureclasses:
                            fc_path = os.path.join(arcpy.env.workspace, dataset, fc)
                            desc = arcpy.Describe(fc_path)

                            # Get the shape type and feature count
                            shape_type = desc.shapeType
                            count = arcpy.GetCount_management(fc_path)[0]

                            # Write to CSV
                            writer.writerow([dir_name, dataset, fc, shape_type, count])

print(f"CSV created successfully at {csv_file}")

How the Script Works:

1. Set Folder and CSV Paths:

   • You need to define the path where your geodatabases are stored (folder_path) and the location of the output CSV file (csv_file).

2. CSV Headers:

   • The script writes a header row into the CSV that includes:

     • GDB Name: The name of the geodatabase.

     • Dataset: The name of the feature dataset (if applicable).

     • Featureclass Name: The name of the feature class.

     • Shape Type: The geometry type (e.g., point, line, polygon).

     • Count: The number of features in the feature class.

3. Directory Traversal:

   • The os.walk() function walks through the folder that contains the geodatabases. If a directory ends with .gdb, it processes that geodatabase.

4. Standalone Feature Classes:

   • The script first lists standalone feature classes (those not in any dataset) within each geodatabase and gets their shape type and feature count.

5. Feature Datasets:

   • If feature datasets exist in the geodatabase, the script processes each feature class within the dataset, retrieves the metadata, and writes the information to the CSV.

6. Writing to CSV:

   • For each feature class, the script writes a new row in the CSV with the collected metadata, including the shape type and count.

Benefits of Using This Script:

• Automation: It automatically collects and exports metadata for all feature classes in a geodatabase, eliminating the need for manual tracking.

• Documentation: The script generates a well-structured CSV that can be used for documentation or reports.

• Batch Processing: Whether you have a few geodatabases or hundreds, this script handles them all in a batch process, saving you valuable time.

• Versatility: You can easily modify the script to capture other metadata or make it work with different data sources.

Conclusion:

Managing geospatial data across multiple geodatabases can become a daunting task without the right tools. By automating the process of collecting feature class metadata and exporting it into a CSV, you can gain deeper insights into your datasets with minimal effort. This script, leveraging ArcPy and Python, ensures that you can process large amounts of data efficiently and keep track of important details such as feature counts, shape types, and more.

Feel free to adjust the file paths and adapt the script to fit your specific project needs. This tool is perfect for data management, quality assurance, and reporting tasks!

 

Automating Field Updates in Geodatabase Feature Classes with Python and ArcPy

In this blog post, I’ll walk you through a Python script that automates the process of updating field types within feature classes in Esri file geodatabases. This solution uses ArcPy, which is part of the ArcGIS API for Python, to streamline data management tasks—perfect for anyone managing large geodatabases and needing to update multiple field types efficiently.

Use Case

Imagine you have several feature classes within your geodatabase, and some of the field types need to be changed—perhaps from integer to string, or altering field length to meet new specifications. Manually updating these fields could take a lot of time, especially if there are many geodatabases and feature classes. That's where automation comes in!

This script reads a CSV file containing the required field updates and applies them across all geodatabases in a specified directory. It ensures that the updates are logged so you can track what was modified or if any errors occurred.

Script Breakdown

Below is the Python script that automates the field update process. It updates field types based on data in a CSV file and provides a log of changes.

Code:

python
import arcpy
import csv
import os

# Input folder containing geodatabases
folder_path = r"PATH_TO_YOUR_GEODATABASES_FOLDER"  # Example: r"C:\path\to\your\geodatabases"
# Input CSV file with updated field information
input_csv = r"PATH_TO_YOUR_CSV_FILE"  # Example: r"C:\path\to\your\fields_to_update.csv"
# Temporary output CSV file
temp_csv = r"PATH_TO_TEMP_CSV_FILE"  # Example: r"C:\path\to\temp_updated_fields.csv"

# Read the CSV file into a list
csv_data = []
with open(input_csv, 'r', newline='', encoding='utf-8') as csvfile:
    reader = csv.DictReader(csvfile)
    csv_headers = reader.fieldnames + ['Update Status']  # Add a new column for update status
    for row in reader:
        row['Update Status'] = 'Not Processed'  # Initialize with default status
        csv_data.append(row)

# Iterate through all items in the folder
for item in os.listdir(folder_path):
    gdb_path = os.path.join(folder_path, item)

    # Check if the item is a geodatabase
    if os.path.isdir(gdb_path) and gdb_path.endswith(".gdb"):
        # Set the workspace to the current geodatabase
        arcpy.env.workspace = gdb_path
        datasets = arcpy.ListDatasets(feature_type='feature') or ['']  # Include standalone feature classes

        for dataset in datasets:
            # List feature classes within each dataset
            feature_classes = arcpy.ListFeatureClasses(feature_dataset=dataset)

            for feature_class in feature_classes:
                # Get full path to feature class
                feature_class_path = f"{gdb_path}\\{dataset}\\{feature_class}" if dataset else f"{gdb_path}\\{feature_class}"

                # Describe fields in the feature class
                fields = arcpy.ListFields(feature_class_path)

                for field in fields:
                    # Check if field exists in the CSV data
                    for csv_row in csv_data:
                        csv_dataset = csv_row['Feature dataset']
                        csv_feature_class = csv_row['Feature class']
                        csv_field = csv_row['Field']

                        # Match dataset, feature class, and field
                        if (
                            (csv_dataset == dataset if dataset else "Standalone") and
                            csv_feature_class == feature_class and
                            csv_field == field.name
                        ):
                            # Attempt to alter the field data type
                            try:
                                arcpy.management.AlterField(
                                    in_table=feature_class_path,
                                    field=field.name,
                                    field_type="TEXT",
                                    field_length=150
                                )
                                csv_row['Update Status'] = 'Updated'
                                print(f"Field {field.name} updated to String (150 characters).")
                            except Exception as e:
                                csv_row['Update Status'] = f"Failed: {str(e)}"
                                print(f"Failed to modify field {field.name}: {e}")

# Write updated CSV data back to a new file
with open(temp_csv, 'w', newline='', encoding='utf-8') as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=csv_headers)
    writer.writeheader()
    writer.writerows(csv_data)

# Replace original CSV with updated one
os.replace(temp_csv, input_csv)

print(f"Field modification process completed. Updates logged in {input_csv}.")

How the Script Works:

1. Input Files:

   • folder_path: The path to the folder that contains your geodatabases.

   • input_csv: The CSV file that holds the required updates (i.e., dataset name, feature class, field name, and new field type).

   • temp_csv: A temporary CSV file where the updated data will be written.

2. Reading the CSV: The script reads the input CSV file and adds an "Update Status" column. This helps keep track of which fields were successfully updated and which ones encountered issues.

3. Iterating Through Geodatabases: The script checks the folder for .gdb files and sets the workspace for each geodatabase. It processes all datasets and feature classes within each geodatabase.

4. Updating Fields:

   • The script matches each field in the feature class against the data from the CSV file.

   • For any matching fields, the script attempts to update the field data type to TEXT with a length of 150.

   • If the update is successful, it updates the status in the CSV to "Updated." If it fails, it logs the error and updates the status to "Failed."

5. Writing Back the CSV: After processing all geodatabases, the script writes the updated CSV data to a new temporary CSV file and replaces the original CSV with the updated one.

Benefits of Using This Script:

• Automation: This saves a tremendous amount of time if you need to make the same changes across multiple geodatabases.

• Logging: The script maintains a log of updates, so you can easily track which fields were updated successfully and which ones failed.

• Scalability: Whether you have 10 or 100 geodatabases, this script can handle large datasets and perform bulk updates without manual intervention.

Conclusion:

Managing geospatial data can be a complex task, especially when working with large geodatabases. Automating repetitive tasks like field type updates not only improves efficiency but also reduces the risk of human error. Using ArcPy and Python, this script simplifies a process that would otherwise take hours and allows you to focus on more critical tasks.

Feel free to customize the script for your specific needs. You can easily modify the field types, field lengths, or the fields you wish to update.

Automating Feature Class Cleanup in Geodatabases using ArcPy

 

🧹 Automating Feature Class Cleanup in Geodatabases using ArcPy

In GIS data management, maintaining a clean and organized geodatabase (GDB) is essential to ensure data efficiency and reduce clutter. When working with large datasets, it’s common to only need a subset of feature classes. In this blog post, I'll show you how to automate the process of deleting unnecessary feature classes from a geodatabase using ArcPy.

This script allows you to specify which feature classes to keep, and it automatically deletes any feature classes that are not on the list. It’s an efficient way to streamline your geodatabase and avoid manual cleanup.

How It Works:

1. Workspace Setup: The script starts by specifying the workspace, which is the path to your GDB.

2. Feature Classes to Keep: A list of feature class names is provided. The script will only retain these feature classes, and all others will be deleted.

3. Feature Class Processing: The script first checks for feature classes at the root level of the GDB, then iterates through any feature datasets (sub-gdbs) to ensure all relevant feature classes are cleaned up.

4. Cleanup: It loops through each feature class in the specified GDB and compares its name against the list of feature classes to keep. If the feature class is not on the list, it will be deleted.

Code:

python
import arcpy
import os

# === INPUT ===
gdb_path = r"C:\Users\Testuser\Downloads\MPDA_MigratedData_25_03\Datamodel.gdb"
featureclasses_to_keep = [
    "LandCover_Area", "Planned_Land_Cover", "Existing_Land_Use", "Population_Settlements", "Parcels",
    "Proposed_Parcel", "Seaport", "Airports", "Bus_Stop_Shelters", "Planned_Roads", "Planned_Metro_Lines",
    "Proposed_Railroads", "Railway_Proposed", "Railways", "Road_Center_Line", "Railway_Facilities", 
    "Railway_Station", "Planned_Metro_Stations", "Bus_Station", "Oil_Facilities", "Oil_Pipes", "Communication_Facility", 
    "Communication_Cables", "Electrical_Cables", "Electrical_Facilities", "Water_Pipes", "Water_Facilities", 
    "Stormwater_Facilities", "Stormwater_Conduits", "Stormwater_Inlets", "Sewerage_Pipes", "Sewerage_Facilities"
]  # Example feature classes to retain (Update with your actual list)
featureclasses_to_keep = [fc.lower() for fc in featureclasses_to_keep]
# Set workspace
arcpy.env.workspace = gdb_path

# Function to delete unwanted feature classes from a given workspace
def clean_featureclasses(workspace, fc_keep_list):
    arcpy.env.workspace = workspace
    fcs = arcpy.ListFeatureClasses()

    for fc in fcs:
        if fc.lower() not in fc_keep_list:
            print(f"Deleting feature class: {fc}")
            arcpy.Delete_management(fc)

# === PROCESS ROOT LEVEL ===
clean_featureclasses(gdb_path, featureclasses_to_keep)

# === PROCESS FEATURE DATASETS ===
datasets = arcpy.ListDatasets(feature_type='feature')

if datasets:
    for ds in datasets:
        ds_path = os.path.join(gdb_path, ds)
        clean_featureclasses(ds_path, featureclasses_to_keep)

print("Cleanup completed successfully.")

Explanation of the Code:

1. Input and Workspace Setup:

   • The gdb_path is set to the location of your File Geodatabase (GDB).

   • A list called featureclasses_to_keep contains the names of feature classes you want to retain in your GDB. These names are converted to lowercase for case-insensitive matching.

2. Function to Clean Feature Classes:

   • The function clean_featureclasses iterates through all feature classes in a specified workspace (root or dataset).

   • It checks if the feature class name exists in the featureclasses_to_keep list. If not, it deletes the feature class.

3. Processing Root Level:

   • The script first cleans up feature classes at the root level of the GDB.

4. Processing Feature Datasets:

   • If there are feature datasets (sub-gdbs), the script recursively processes them, cleaning up any feature classes that do not need to be kept.

Why Use This Script?

• Automation: This script automates the tedious task of cleaning up unwanted feature classes, saving time and reducing the chance of human error.

• Efficiency: It ensures that only the feature classes you need are retained, optimizing the size and organization of your geodatabases.

• Scalability: You can apply this method to multiple geodatabases, making it ideal for large-scale projects with numerous GDBs.

Conclusion:

By using this ArcPy script, you can automate the cleanup of feature classes in your GDBs, ensuring that only the necessary data remains, and reducing clutter in your GIS databases. It’s an essential tool for GIS analysts and developers working with large datasets.

Batch Field Deletion from Feature Classes in Multiple GDBs (ArcPy)

🔄 Batch Field Deletion from Feature Classes in Multiple GDBs (ArcPy)

In GIS data management, it is often necessary to remove redundant or obsolete fields from feature classes to streamline workflows and ensure data consistency. This ArcPy script automates the process of deleting specified fields from all feature classes across multiple File Geodatabases (GDBs).

This solution is particularly useful when dealing with multiple GDBs in large projects, helping to maintain a clean and manageable data model without manual intervention for each feature class.

How the Script Works:

1. Directory Setup: The script starts by defining the folder that contains all the GDBs.

2. Field List: A list of field names that need to be deleted is defined. In this case, the fields to be deleted include metadata fields like "OriginalName", "OriginalPath", and "DIA_ID".

3. Iterating through GDBs: The script loops through the specified folder, identifying which subfolders are GDBs based on their .gdb extension.

4. Feature Class Processing: Inside each GDB, the script processes the feature classes, checking for the existence of the fields defined for deletion.

5. Field Deletion: If any of the specified fields exist in the feature class, they are deleted. If the field is missing, a message is printed, and the script continues processing.

6. Completion: Once all feature classes in all GDBs have been processed, the script outputs a message confirming the completion of the field deletion process.

This automated solution saves significant time and effort, especially when managing large-scale GIS projects with numerous geodatabases.

The Code:

python
import arcpy
import os

# Define the directory containing the file geodatabases
gdb_folder = r'C:\Users\Testuser\OneDrive - GPC Global Information Solutions LLC\Projects\MPDA\DataModel\Schema_V2'

# Define the fields to be deleted
fields_to_delete = ["ORIGINALNAME","ORIGINALPATH", "DIA_ID", "DIA_DATE"]#["EMIRATE_ID"]

# Iterate through each folder in the specified directory
for folder in os.listdir(gdb_folder):
    # Check if the folder has a .gdb extension (i.e., it's a file geodatabase)
    if folder.endswith(".gdb"):
        gdb_path = os.path.join(gdb_folder, folder)
        arcpy.env.workspace = gdb_path

        # List all datasets within the geodatabase
        datasets = arcpy.ListDatasets("*", "Feature") or [None]

        for dataset in datasets:
            # Set the workspace to the dataset if it exists, else to the geodatabase
            if dataset:
                feature_classes = arcpy.ListFeatureClasses("*", "", dataset)
            else:
                feature_classes = arcpy.ListFeatureClasses()

            # Iterate through each feature class
            for fc in feature_classes:
                print(f"Working on Feature Class: {fc}")

                # Iterate through each field to be deleted
                for field_name in fields_to_delete:
                    # Check if the field exists
                    fields = arcpy.ListFields(fc, field_name)
                    if fields:
                        # Delete the field
                        arcpy.DeleteField_management(fc, field_name)
                        print(f"Deleted field {field_name} from {fc}")
                    else:
                        print(f"Field {field_name} does not exist in {fc}")

print("Field deletion process completed.")

Key Points to Remember:

• The script automatically iterates through all the GDBs in a specified folder, processing each feature class.

• It checks for specific fields (such as "OriginalName", "OriginalPath", etc.) and deletes them if they exist.

• If a field does not exist in a feature class, it simply skips that field and moves on to the next one.

• The script is highly efficient for projects involving large datasets, reducing manual overhead and ensuring data consistency.

🗂️ Batch Processing and Copying Feature Classes from CSV (ArcPy Script)

 

🗂️ Batch Processing and Copying Feature Classes from CSV (ArcPy Script)

This Python script uses ArcPy to automate the process of copying feature classes from various spatial data sources to a geodatabase, using metadata from a CSV file. The script performs several validations, sanitizes names, checks paths, and ensures that all relevant data (e.g., fields like OriginalName, DIA_ID, DIA_Date) are added to the copied feature classes.

---

🔑 Key Features:

• Sanitize Feature Class Names: Automatically removes unwanted characters from feature class names.

• Validate Feature Class Paths: Checks if paths are accessible, not exceeding the Windows path limit, and whether the data format is supported.

• Spatial Data Check: Ensures that only spatial data is copied (non-spatial data is skipped).

• Add Metadata: Adds relevant metadata fields like OriginalName, OriginalPath, DIA_ID, and DIA_Date to the feature classes.

• Handle Errors Gracefully: Catches and logs any errors during the copy process, providing detailed messages for debugging.

---

🧑‍💻 Python Script:

python
import arcpy
import pandas as pd
import os
import re
import sys
from datetime import datetime

# Ensure the default encoding is set to UTF-8
reload(sys)
sys.setdefaultencoding('utf-8')

# Define paths
csv_file = u'C:\MPDA\GDB_Extraction\Environmental_Vector.csv'
output_folder = u'C:\MPDA\GDB_Extraction\Enviromental vector'

# List to capture failed paths
failed_paths = []

# Function to sanitize feature class names
def sanitize_name(name):
    sanitized = re.sub(r'[^a-zA-Z0-9_\u0600-\u06FF]', '_', name)
    return sanitized

# Function to check if a field exists in a feature class
def field_exists(feature_class, field_name):
    if arcpy.Exists(feature_class):
        fields = [f.name for f in arcpy.ListFields(feature_class)]
        return field_name in fields
    else:
        print(u"Feature class does not exist: {}".format(feature_class))
        return False

# Function to parse the date with multiple formats
def parse_date(date_str):
    formats = ['%m/%d/%Y', '%Y/%m/%d', '%d/%m/%Y']
    for fmt in formats:
        try:
            return datetime.strptime(date_str, fmt)
        except ValueError:
            continue
    return None

# Function to check if a path is valid and supported
def is_supported_path(path):
    if len(path) > 260:
        print(u"Path exceeds the maximum length allowed by Windows: {}".format(path))
        failed_paths.append(path)
        return False
    if not arcpy.Exists(path):
        print(u"Path does not exist or is inaccessible: {}".format(path))
        failed_paths.append(path)
        return False
    if path.lower().endswith('.mdb'):
        print(u"Unsupported file format (MDB): {}".format(path))
        failed_paths.append(path)
        return False
    return True

# Function to check if the feature class contains spatial data
def contains_spatial_data(feature_class):
    desc = arcpy.Describe(feature_class)
    if hasattr(desc, "shapeType"):
        return True
    else:
        print(u"Non-spatial data encountered: {}".format(feature_class))
        failed_paths.append(feature_class)
        return False

# Read the CSV file into a pandas DataFrame
df = pd.read_csv(csv_file, encoding='utf-8')

# Print the column names to check if they exist
print(u"Column names in CSV:", df.columns)

# Strip any leading/trailing spaces from column names
df.columns = df.columns.str.strip()

# Group the rows in the CSV by the 'Theme' field
grouped = df.groupby('Theme')

# Iterate over each unique theme
for theme, group in grouped:
    # Create a sanitized name for the GDB
    theme_sanitized = sanitize_name(theme)
    theme_gdb = os.path.join(output_folder, u"{}.gdb".format(theme_sanitized))

    # Ensure the output GDB for the theme exists
    if not arcpy.Exists(theme_gdb):
        arcpy.CreateFileGDB_management(output_folder, u"{}.gdb".format(theme_sanitized))

    # Iterate through each row in the grouped DataFrame for this theme
    for index, row in group.iterrows():
        input_path = row['Shape']
        featureclass_name = os.path.basename(input_path).split('.')[0]
        sanitized_name = sanitize_name(featureclass_name)
        output_featureclass = os.path.join(theme_gdb, sanitized_name)

        # Check if the path is valid and supported
        if is_supported_path(input_path):
            # Check if the feature class contains spatial data
            if contains_spatial_data(input_path):
                # Check for existing feature class with the same name and create a unique name
                counter = 1
                while arcpy.Exists(output_featureclass):
                    output_featureclass = os.path.join(theme_gdb, u"{}_{}".format(sanitized_name, counter))
                    counter += 1

                # Debugging output
                print(u"Copying from {} to {}".format(input_path, output_featureclass))

                try:
                    # Copy the feature class to the output GDB
                    arcpy.CopyFeatures_management(input_path, output_featureclass)
                    print(u"Successfully copied feature class to {}".format(output_featureclass))
                except arcpy.ExecuteError:
                    print(u"Failed to copy feature class to {}".format(output_featureclass))
                    print(arcpy.GetMessages(2))
                    failed_paths.append(input_path)
                    continue

                # Check if the copied feature class exists before proceeding
                if arcpy.Exists(output_featureclass):
                    # Add fields for original name, path, DIA_ID, and DIA_Date if they do not already exist
                    if not field_exists(output_featureclass, "OriginalName"):
                        arcpy.AddField_management(output_featureclass, "OriginalName", "TEXT", field_length=255)
                    if not field_exists(output_featureclass, "OriginalPath"):
                        arcpy.AddField_management(output_featureclass, "OriginalPath", "TEXT", field_length=1000)
                    if not field_exists(output_featureclass, "DIA_ID"):
                        arcpy.AddField_management(output_featureclass, "DIA_ID", "TEXT")
                    if not field_exists(output_featureclass, "DIA_Date"):
                        arcpy.AddField_management(output_featureclass, "DIA_Date", "DATE")

                    # Update the new fields with the original name, path, and other CSV data
                    with arcpy.da.UpdateCursor(output_featureclass, ["OriginalName", "OriginalPath", "DIA_ID", "DIA_Date"]) as cursor:
                        for cursor_row in cursor:
                            cursor_row[0] = featureclass_name
                            cursor_row[1] = row['File Path']  # Fill OriginalPath with the value from the File Path column

                            cursor_row[2] = row['ID'] if 'ID' in df.columns else None

                            if 'DIA_Date' in df.columns and pd.notnull(row['DIA_Date']):
                                date_value = parse_date(row['DIA_Date'])
                                if date_value:
                                    cursor_row[3] = date_value
                                else:
                                    print(u"Error parsing date for row {}: Invalid date format '{}'".format(index, row['DIA_Date']))
                                    cursor_row[3] = None
                            else:
                                cursor_row[3] = None

                            cursor.updateRow(cursor_row)
                else:
                    print(u"Copied feature class does not exist: {}".format(output_featureclass))
                    failed_paths.append(input_path)
            else:
                print(u"Skipping non-spatial data: {}".format(input_path))
        else:
            print(u"Skipping unsupported or inaccessible path: {}".format(input_path))

# Print the list of failed paths
if failed_paths:
    print("\nThe following file paths failed during the process:")
    for path in failed_paths:
        print(path)
else:
    print("\nAll file paths were processed successfully.")

print(u"Process completed successfully.")

---

🧑‍💻 How It Works:

1. CSV Input: The script reads metadata from a CSV file.

2. Feature Class Copying: It copies the feature classes from their input paths to output GDBs, ensuring data validity.

3. Field Management: Adds necessary fields and populates them with metadata from the CSV file.

4. Error Handling: Logs failed paths for further review.
   
   

   Sample Input CSV:

   Shape	File Path	Theme	ID	DIA_Date
   C:\Data\Environmental\Area1.shp	C:\Data\Environmental\Area1.shp	Environmental	1001	01/05/2020
   C:\Data\Environmental\Area2.shp	C:\Data\Environmental\Area2.shp	Environmental	1002	03/12/2021
   C:\Data\Environmental\Water.shp	C:\Data\Environmental\Water.shp	Water	1003	11/22/2022
   C:\Data\Environmental\Soil.shp	C:\Data\Environmental\Soil.shp	Environmental	1004	06/14/2020
   C:\Data\WaterResources\River.shp	C:\Data\WaterResources\River.shp	Water	1005	08/18/2021
   C:\Data\WaterResources\Lake.shp	C:\Data\WaterResources\Lake.shp	Water	1006	09/01/2019

   CSV Column Explanation:

   • Shape: The full path to the input feature class (Shapefile or other formats).

   • File Path: The location of the file on your system, which may be used as additional metadata.

   • Theme: The thematic grouping or category for the feature class (e.g., Environmental, Water, etc.).

   • ID: A unique identifier for the feature class, which could correspond to specific metadata or other cataloging information.

   • DIA_Date: The date associated with the feature class data (e.g., the date it was created, modified, or captured).

   ---

   How It Relates to the Script:

   1. Shape: The path to the spatial data file (Shapefile or feature class).

   2. File Path: Used to store the file path as metadata in the output feature classes.

   3. Theme: Grouped by the script into separate geodatabases for organization.

   4. ID: Added to the feature class as a field if present.

   5. DIA_Date: Added as a date field and parsed using multiple formats.
      
      

      ✅ Conclusion:

      This script automates copying spatial data to organized geodatabases, ensuring all fields and metadata are correctly updated while handling any issues gracefully.

🗂️ Group and Convert CSV Files to Excel (Python)

 

🗂️ Group and Convert CSV Files to Excel (Python)

This script will:

1. Group CSV files based on their naming convention (by prefix and category).

2. Read each CSV file, which is delimited by | (pipe character).

3. Combine data from each group into a single Excel file with separate sheets for each category.

4. Save the final output as Excel files organized by their prefix.

---

🔑 Key Steps:

• File Grouping: It identifies and groups CSV files by their prefix (e.g., "Area", "Basemap").

• Merging Data: All CSVs in each group are combined into one Excel sheet for that category.

• Excel Output: The final grouped data is written into Excel files, with each category in a separate sheet.

---

🧑‍💻 Python Script:

python
import os
import pandas as pd

# Define the input folder containing all CSV files
input_folder = r"C:\Users\Testuser\OneDrive - GPC Global Information Solutions LLC\Projects\MPDA\DataModel\Schema_V2\CSV\Schema"
output_folder = r"C:\Users\Testuser\OneDrive - GPC Global Information Solutions LLC\Projects\MPDA\DataModel\Schema_V2\CSV\Schema\SchemaExcel"

# Create the output folder if it doesn't exist
os.makedirs(output_folder, exist_ok=True)

# Dictionary to group files by their prefix (e.g., "Area", "Basemap")
grouped_files = {}

# Group files based on prefix
for csv_file in os.listdir(input_folder):
    if csv_file.endswith(".csv"):
        # Split filename to get prefix and category
        parts = csv_file.split("_")
        if len(parts) < 2:
            continue  # Skip files that don't follow the naming convention

        prefix = parts[0]  # E.g., "Area" or "Basemap"
        category = parts[1].split(".")[0]  # E.g., "Domain" or "Schema"

        # Group files under their prefix and category
        if prefix not in grouped_files:
            grouped_files[prefix] = {}
        if category not in grouped_files[prefix]:
            grouped_files[prefix][category] = []
        grouped_files[prefix][category].append(os.path.join(input_folder, csv_file))

# Create Excel files for each prefix group
for prefix, categories in grouped_files.items():
    output_file = os.path.join(output_folder, f"{prefix}.xlsx")
    writer = pd.ExcelWriter(output_file, engine='openpyxl')

    for category, files in categories.items():
        combined_df = pd.DataFrame()  # Initialize empty DataFrame
        for file in files:
            try:
                # Read CSV with "|" as the delimiter
                df = pd.read_csv(file, encoding='utf-8', delimiter='|', on_bad_lines='skip')
                combined_df = pd.concat([combined_df, df], ignore_index=True)
            except Exception as e:
                print(f"Error processing file {file}: {e}")

        # Write to a sheet named after the category
        combined_df.to_excel(writer, sheet_name=category[:31], index=False)  # Sheet names max 31 chars

    # Save and close the Excel writer
    writer.close()
    print(f"Excel file created: {output_file}")

---

📋 Key Features:

1. Grouping: Organizes CSV files based on the prefix (e.g., "Area", "Basemap").

2. Excel Sheets: Each category within a prefix gets its own Excel sheet.

3. Error Handling: It gracefully handles errors when reading malformed CSVs or reading issues with specific files.

4. Efficient Output: All grouped data is consolidated into a clean, organized Excel workbook for each prefix.

---

✅ Conclusion:

This solution provides an efficient method for converting and consolidating multiple CSV files into structured Excel workbooks. It's perfect for organizing large datasets, especially when dealing with a variety of related CSV files that need to be grouped and categorized.