{"id":31591,"date":"2020-09-01T20:12:47","date_gmt":"2020-09-01T20:12:47","guid":{"rendered":"https:\/\/dev.railscarma.com\/data-scraping-in-rails-by-processing-csv\/"},"modified":"2021-06-04T10:22:38","modified_gmt":"2021-06-04T10:22:38","slug":"dataskrapning-i-skenor-genom-att-bearbeta-csv","status":"publish","type":"post","link":"https:\/\/www.railscarma.com\/sv\/blogg\/tekniska-artiklar\/dataskrapning-i-skenor-genom-att-bearbeta-csv\/","title":{"rendered":"Dataskrapning i skenor genom att bearbeta CSV"},"content":{"rendered":"
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

The ruby on rails Ans\u00f6kan om att skrapa l\u00e4nken uppladdad fr\u00e5n CSV-fil och<\/p>\n\n\n\n

hitta f\u00f6rekomsten av l\u00e4nk p\u00e5 en viss sida.<\/p>\n\n\n\n

I applikationen m\u00e5ste anv\u00e4ndaren skicka en csv och en lista \u00f6ver anv\u00e4ndarnas e-post till vilka den analyserade CSV:en kommer att skickas.<\/p>\n\n\n\n

I csv kommer det att finnas tre 2 kolumner:<\/p>\n\n\n\n

  • refferal_link<\/li>
  • home_link<\/li>
  • och det finns v\u00e4rden som nedan<\/li><\/ul>\n\n\n\n

    F\u00f6rst och fr\u00e4mst kommer vi att skapa rails-applikationen<\/p>\n\n\n\n

    $ rails nya scrape_data<\/em><\/strong><\/p>\n\n\n\n

    $ cd scrape_data<\/em><\/strong><\/p>\n\n\n\n

    Sedan genererar vi UploadCsv-modulen, k\u00f6r kommandot nedan<\/p>\n\n\n\n

    $ skenor g st\u00e4llning UploadCsv generated_csv:string csv_file:string<\/em><\/strong><\/p>\n\n\n\n

    Det kommer att skapa alla n\u00f6dv\u00e4ndiga modeller, kontroller och migrationer f\u00f6r csv_file<\/p>\n\n\n\n

    D\u00e5 b\u00f6rjar vi med att f\u00f6rst ladda upp filen i DB<\/p>\n\n\n\n

    ers\u00e4tt nedanst\u00e5ende kod i filer  app\/views\/upload_csvs\/_form.html.erb<\/em><\/strong><\/p>\n\n\n\n

    vi har lagt till koden nedan f\u00f6r att ladda upp filen<\/p>\n\n\n\n

    <%= form_with(modell: upload_csv, lokal: true) g\u00f6r |form| %><\/p>\n\n\n\n

      <% om upload_csv.errors.any? %><\/p>\n\n\n\n

        <div id="”error_explanation”"><\/p>\n\n\n\n

          <h2><%= pluralize(upload_csv.errors.count, “error”) %> förbjöd denna upload_csv från att sparas:<\/h2><\/p>\n\n\n\n

         

      <\/p>\n\n\n\n

              <% upload_csv.errors.full_messages.each do |message| %><\/p>\n\n\n\n

               

    • <%= meddelande %><\/li><\/p>\n\n\n\n

              <% slut %><\/p>\n\n\n\n

            <\/ul><\/p>\n\n\n\n

          <\/div>.<\/p>\n\n\n\n

        <% slut %><\/p>\n\n\n\n

        <div class="”field”"><\/p>\n\n\n\n

          <%= form.label :csv_file %><\/p>\n\n\n\n

          <%= form.filf\u00e4lt :csv_fil %><\/p>\n\n\n\n

        <\/div>.<\/p>\n\n\n\n

        <div class="”actions”"><\/p>\n\n\n\n

          <%= form.submit %><\/p>\n\n\n\n

        <\/div>.<\/p>\n\n\n\n

      <% slut %><\/p>\n\n\n\n

      Sedan l\u00e4gger vi till p\u00e4rlan f\u00f6r att ladda upp en csv_file<\/p>\n\n\n\n

      l\u00e4gg till raden nedan i gem-filen<\/p>\n\n\n\n

      gem 'carrierwave', '~> 2.0'<\/p>\n\n\n\n

      $-paketinstallation<\/em><\/strong><\/p>\n\n\n\n

      Sedan skapar vi uppladdningsprogrammet i carrierwave<\/p>\n\n\n\n

      $-skenor genererar uppladdningsavatar<\/em><\/strong><\/p>\n\n\n\n

      vi kommer att bifoga uppladdaren i modellen<\/p>\n\n\n\n

      app\/models\/upload_csv.rb<\/em><\/strong><\/p>\n\n\n\n

      klass UploadCsv < ApplicationRecord<\/p>\n\n\n\n

        mount_uploader :csv_file, AvatarUploader<\/p>\n\n\n\n

      slutet<\/p>\n\n\n\n

      innan du g\u00e5r vidare, kontrollera bara att din applikation fungerar<\/p>\n\n\n\n

      k\u00f6r under kommandon<\/p>\n\n\n\n

      $ rake db:create db:migrate<\/em><\/strong><\/p>\n\n\n\n

      uppdatera rutterna<\/p>\n\n\n\n

      Rails.application.routes.draw g\u00f6r<\/p>\n\n\n\n

        resurser :upload_csvs<\/p>\n\n\n\n

        root 'upload_csvs#index'<\/p>\n\n\n\n

      slutet<\/p>\n\n\n\n

      $ skenor s<\/em><\/strong><\/p>\n\n\n\n

      Sedan skapar vi ett jobb f\u00f6r att l\u00e4sa CSV-filen och skrapa l\u00e4nken fr\u00e5n den<\/p>\n\n\n\n

      och den genererade filen kommer att sparas i kolumnen genered_csv i den posten<\/p>\n\n\n\n

      f\u00f6r att skapa jobbet kommer vi att g\u00f6ra som nedan<\/p>\n\n\n\n

      $-skenor genererar jobb <\/em>generera_csv<\/em><\/strong><\/p>\n\n\n\n

      l\u00e4gg till p\u00e4rlan nedan och k\u00f6r paketinstallationen<\/p>\n\n\n\n

      p\u00e4rla 'httparty'<\/p>\n\n\n\n

      p\u00e4rla 'nokogiri'<\/p>\n\n\n\n

      d\u00e5 ers\u00e4tter vi koden med nedan<\/p>\n\n\n\n

      klass GenrateCsvJob < ApplicationJob<\/p>\n\n\n\n

        queue_as :default<\/p>\n\n\n\n

        def perform(upload_csv)<\/p>\n\n\n\n

          processed_csv(upload_csv)<\/p>\n\n\n\n

          fil = Tempfile.open([\u201c#{Rails.root}\/public\/generated_csv\u201d, '.csv']) till |csv|<\/p>\n\n\n\n

            csv << %w[referal_link home_link count]<\/p>\n\n\n\n

            @new_array.each g\u00f6r |new_array|<\/p>\n\n\n\n

              csv << new_array<\/p>\n\n\n\n

            slutet<\/p>\n\n\n\n

            file = "#{Rails.root}\/public\/product_data.csv"<\/p>\n\n\n\n

            headers = ['referal_link', 'home_link', 'count']<\/p>\n\n\n\n

            file = CSV.open(file, 'w', write_headers: true, headers: headers) do |writer|<\/p>\n\n\n\n

              @new_array.each g\u00f6r |new_array|<\/p>\n\n\n\n

                f\u00f6rfattare << new_array<\/p>\n\n\n\n

              slutet<\/p>\n\n\n\n

              upload_csv.update(generated_csv: fil)<\/p>\n\n\n\n

            slutet<\/p>\n\n\n\n

          slutet<\/p>\n\n\n\n

          NotificationMailer.send_csv(upload_csv).deliver_now! om @new_array.present?<\/p>\n\n\n\n

          1TP5M\u00e5ste generera f\u00f6rs\u00e4ndelsen och f\u00f6lja stegen f\u00f6r f\u00f6rs\u00e4ndelsen<\/p>\n\n\n\n

        slutet<\/p>\n\n\n\n

        # Metod f\u00f6r att f\u00e5 l\u00e4nkr\u00e4kningen och lagrar i arrayen<\/p>\n\n\n\n

        def processed_csv(upload_csv)<\/p>\n\n\n\n

          @new_array = []<\/p>\n\n\n\n

          CSV.foreach(upload_csv.csv_file.path, headers: true, header_converters: :symbol) g\u00f6r |rad|<\/p>\n\n\n\n

            rad_karta = rad.till_h<\/p>\n\n\n\n

            page = HTTParty.get(row_map[:refferal_link])<\/p>\n\n\n\n

            page_parse = Nokogiri::HTML(page)<\/p>\n\n\n\n

            link_array = page_parse.css('a').map { |l\u00e4nk| l\u00e4nk['href'] }<\/p>\n\n\n\n

            link_array_group = link_array.group_by(&:itself).map { |k, v| [k, v.l\u00e4ngd] }.till_h<\/p>\n\n\n\n

            @new_array.push([radkarta[:refferal_l\u00e4nk], radkarta[:heml\u00e4nk], (l\u00e4nkarraygrupp[radkarta[:heml\u00e4nk]]).to_s])<\/p>\n\n\n\n

          slutet<\/p>\n\n\n\n

        slutet<\/p>\n\n\n\n

      slutet<\/p>\n\n\n\n

      Sedan bifogar vi jobbet after_create av upload_csvs och vi l\u00e4gger till valideringen f\u00f6r csv_file require<\/p>\n\n\n\n

       v\u00e4nligen uppdatera koden f\u00f6r  app\/models\/upload_csv.rb<\/em><\/strong><\/p>\n\n\n\n

      klass UploadCsv < ApplicationRecord<\/p>\n\n\n\n

        mount_uploader :csv_file, AvatarUploader<\/p>\n\n\n\n

        after_create :processed_csv<\/p>\n\n\n\n

        def processed_csv<\/p>\n\n\n\n

          GenrateCsvJob.perform_later(self)<\/p>\n\n\n\n

        slutet<\/p>\n\n\n\n

      slutet<\/p>\n\n\n\n

      kontrollera sedan efter uppladdning av filen din scrape-genererade fil kommer att uppdateras du kan kontrollera genererad csv<\/p>\n\n\n\n

      inuti  \/scrape_data\/public\/product_data.csv<\/em><\/strong><\/p>\n\n\n\n

      vi kan skicka via e-post genom att anv\u00e4nda instruktionerna nedan<\/p>\n\n\n\n

      F\u00f6rst kommer vi att generera f\u00f6rs\u00e4ndelsen<\/p>\n\n\n\n

      $-skenor genererar mailer NotificationMailer<\/em><\/strong><\/p>\n\n\n\n

      uppdatera koden f\u00f6r app\/mailers\/notification_mailer.rb<\/em><\/strong><\/p>\n\n\n\n

        def send_csv(upload_csv)<\/p>\n\n\n\n

          @greeting = 'Hej'<\/p>\n\n\n\n

          attachments['parsed.csv'] = File.read(upload_csv.generated_csv)<\/p>\n\n\n\n

          mail(till: "sample@gmail.com", \u00e4mne: 'CSV har analyserats framg\u00e5ngsrikt.')<\/p>\n\n\n\n

        slutet<\/p>\n\n\n\n

      slutet<\/p>\n\n\n\n

      v\u00e4nligen konfigurera e-postkonfigurationen ocks\u00e5 config\/environments\/development.rb eller production.rb<\/em><\/strong><\/p>\n\n\n\n

      l\u00e4gg till nedanst\u00e5ende rader i filen<\/p>\n\n\n\n

      config.action_mailer.default_url_options = { v\u00e4rd: 'https:\/\/sample-scrape.herokuapp.com\/' }<\/p>\n\n\n\n

      config.action_mailer.delivery_method = :smtp<\/p>\n\n\n\n

      config.action_mailer.smtp_settings = {<\/p>\n\n\n\n

        anv\u00e4ndarnamn: 'exempel@gmail.com',<\/p>\n\n\n\n

        l\u00f6senord: '*******123456',<\/p>\n\n\n\n

        dom\u00e4n: 'gmail.com',<\/p>\n\n\n\n

        adress: 'smtp.gmail.com',<\/p>\n\n\n\n

        port: '587',<\/p>\n\n\n\n

        autentisering: :plain<\/p>\n\n\n\n

      }<\/p>\n\n\n\n

      config.action_mailer.raise_delivery_errors = falskt<\/p>\n\n\n\n

      och uppdatera vyn ocks\u00e5 app\/views\/notification_mailer\/send_csv.html.erb<\/em><\/strong><\/p>\n\n\n\n

      <h1>CSV har bearbetats, tack!<\/h1><\/p>\n\n\n\n

      <p><\/p>\n\n\n\n

        <%= @h\u00e4lsning %>, kontrollera bilagan f\u00f6r att f\u00e5 e-postmeddelandet<\/p>\n\n\n\n

      <\/p><\/p>\n\n\n\n

      Tack !<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t

      Prenumerera f\u00f6r de senaste uppdateringarna<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t
      \t\t\t\t\t