How to insert big data on the laravel?

Title: Efficiently Inserting Large Amounts of Data in Laravel Introduction Laravel is an excellent and widely used PHP framework that enables us to develop efficient web applications with ease. However, when it comes to handling large amounts of data, the default approach might not be as effective. In this blog post, we'll explore how to optimize your code for faster insertion of big datasets in Laravel projects. 1. Understand the limitations of Laravel Laravel uses Eloquent ORM (Object-Relational Mapping) to interact with databases and perform queries. While this is efficient on small data sets, it can slow down when handling large amounts of data. The default implementation relies on multiple database transactions that might consume considerable resources over time. 2. Split the data into smaller chunks To optimize the insertion process, we can break up the large data set into smaller subsets. This approach helps to distribute the workload, improving overall performance and resource utilization. You can use Python's pandas library or any other data manipulation tool to split your data into smaller parts. 3. Insert in batches using transactions Once you have your chunks ready, we will insert them using Laravel's database transaction mechanism. This ensures accurate data handling and helps maintain the integrity of your data while improving performance. Here's an example code snippet:

$insert_data = []; 
foreach ($json['value'] as $chunk => $values) {
    // Loop through each subset of values
    foreach ($values as $value) {
        $posting_date = Carbon::parse($value['Posting_Date']);
        $posting_date = $posting_date->format('Y-m-d');
        ....
        // Insert each row into the database using transaction
        DB::beginTransaction();
            try {
                $data = [
                    'item_no'                   => $value['Item_No'],
                    'entry_no'                  => $value['Entry_No'],
                    'document_no'               => $value['Document_No'],
                    'posting_date'              => $posting_date,
                    ....
                ];
                $insert_data[] = $data;
            } catch (Exception $e) {
                DB::rollback(); // Rollback if any error occurs
            }
        $chunk_inserted = $this->batchInsert($insert_data);
        if ($chunk_inserted) {
            DB::commit(); // Commit the insertion if all rows are inserted successfully
        } else {
            DB::rollback(); // Rollback in case of any errors or failure
        }
    }
}

4. Use the batchInsert method for efficient insertions Instead of directly using Laravel's insert() method, we can create a custom batchInsert() method to handle multiple rows at once:

public function batchInsert($insert_data) {
    $rows = count($insert_data);
    $chunk_size = 1000; // Adjust this according to your database limits 
    
    DB::beginTransaction();
        try {
            $counter = 0;
            while ($counter < $rows) {
                $chunk = array_slice($insert_data, $counter, $chunk_size);
                // Insert the chunk in batches using Laravel's insert() method
                \DB::table('items_details')->insert($chunk);
                    if (abs(count($chunk) - $chunk_size) < $chunk_size / 2) {
                        continue;
                    }
            }
            // Insert the remaining rows, if any
            $remaining_rows = count($insert_data) % $chunk_size === 0 ? 0 : $chunk_size - (count($insert_data) % $chunk_size);
            $chunk = array_slice($insert_data, $counter + $remaining_rows);
            // Insert the remaining rows
            \DB::table('items_details')->insert($chunk);
        } catch (Exception $e) {
            DB::rollback();
        }
    if ($rows === 0) {
        return false;
    } else {
        return true;
    }
}

Conclusion By implementing these techniques and optimizing your code, you can significantly improve the performance and efficiency of inserting large amounts of data into Laravel applications. Always test and benchmark your code to ensure it meets your project's specific requirements. Remember that efficient database management is a crucial aspect of any web application development process.