Identifying the problem

The crash happened inside Slots::SyncAllCalendarsService, a service that syncsmultiple calendars concurrently using the Async gem. What made it particularlypuzzling was that switching from Async::Barrier to Thread.new made the errordisappear. This led us down a rabbit hole thinking it was an Async-specificissue.

The stack trace pointed to a strange location:

activerecord (7.2.1.1) lib/arel/visitors/to_sql.rb:898:in `each'  from lib/arel/visitors/to_sql.rb:898:in `each_with_index'  from lib/arel/visitors/to_sql.rb:898:in `inject_join'  from lib/arel/visitors/to_sql.rb:627:in `visit_Arel_Nodes_Or'  ... 1132 levels...

Over a thousand levels deep into visit_Arel_Nodes_Or. That's unusual. Whywould visiting OR nodes in a SQL query cause such deep recursion?

The stack trace mentioned Slots::SyncAllCalendarsService, but that serviceitself had nothing obviously wrong. It was just iterating through calendars andsyncing them. We traced deeper into the code and found the actual culprit inNeeto's Integrations::Icloud::SyncEventsService:

def abandoned_events  pairs = event_params.map { |e| [e[:url], e[:recurrence_id]] }  @_abandoned_events = calendar.events    .where("start_time BETWEEN ? AND ?", start_time, end_time)    .where.not([:url, :recurrence_id] => pairs)end

This code finds "abandoned" events by excluding events that match specific URLand recurrence ID combinations. In production, pairs contained 233 entries.Nothing immediately suspicious, right?

To understand what was happening, we needed to see what Rails was actually doingwith this query. The where.not([:url, :recurrence_id] => pairs) syntax isRails' composite key feature. With 233 pairs, Rails generates SQL like this:

WHERE NOT (  (url = 'url1' AND recurrence_id = 'rec1') OR  (url = 'url2' AND recurrence_id = 'rec2') OR  (url = 'url3' AND recurrence_id = 'rec3') OR  -- ... 230 more OR clauses)

233 OR clauses. That's a lot, but databases can handle it. The problem wasn'tthe SQL itself - it was how Rails built the internal representation of thatquery.

Going into Rails Internals

To really understand what changed, we used bundle open activerecord to look atthe Rails source code. We compared the same code in Rails 7.1.5.2 and 7.2.1.1.

What We Found in Rails 7.1.5.2

In Rails 7.1.5.2, the code that visits OR nodes looked like this:

def visit_Arel_Nodes_Or(o, collector)  stack = [o.right, o.left]  while o = stack.pop    if o.is_a?(Arel::Nodes::Or)      stack.push o.right, o.left    else      visit o, collector      collector << " OR " unless stack.empty?    end  end  collectorend

This is an iterative approach using a manual stack. No matter how deeply nestedthe OR conditions are, this code only uses a single stack frame for thevisit_Arel_Nodes_Or method. It's stack-safe.

What Changed in Rails 7.2.1.1

In Rails 7.2.1.1, the same method became much simpler:

def visit_Arel_Nodes_Or(o, collector)  inject_join o.children, collector, " OR "end

Where inject_join does:

def inject_join(list, collector, join_str)  list.each_with_index do |x, i|    collector << join_str unless i == 0    collector = visit(x, collector)  # Recursive call  end  collectorend

This is a recursive approach. If an OR node contains another OR node, it callsvisit, which calls visit_Arel_Nodes_Or, which calls inject_join again. Therecursion depth grows with the nesting depth of the OR tree.

The Tree Structure Problem

But why is the OR tree so deeply nested? We found the answer in Active Record'sPredicateBuilder#grouping_queries:

def grouping_queries(queries)  if queries.one?    queries.first  else    queries.map! { |query| query.reduce(&:and) }    queries = queries.reduce { |result, query| Arel::Nodes::Or.new([result, query]) }    Arel::Nodes::Grouping.new(queries)  endend

That reduce call is the key. It builds a left-deep nested tree:

Level 1:  Query1 OR Query2 = OR_Node_1Level 2:  OR_Node_1 OR Query3 = OR_Node_2Level 3:  OR_Node_2 OR Query4 = OR_Node_3...Level 232: OR_Node_231 OR Query233 = OR_Node_232

With 233 pairs, this creates 232 levels of nesting. Each level addsapproximately 5 stack frames when Rails traverses it recursively. That's 1,160stack frames just for the query building.

The Async Connection

Why did Thread.new work but Async::Barrier didn't?

We wrote a test script to measure the actual stack depth limits in differentcontexts:

def test_recursion_depth(context_name)  depth = 0  recurse = lambda do |n|    depth = n    recurse.call(n + 1)  end  begin    recurse.call(0)  rescue SystemStackError    return depth  endendthread_depth = nilThread.new do  thread_depth = test_recursion_depth("thread")end.joinputs "Thread recursion limit: ~#{thread_depth} calls"async_depth = nilAsync do  async_depth = test_recursion_depth("async")endputs "Async fiber recursion limit: ~#{async_depth} calls"

When we ran this, the results were revealing:

Thread recursion limit:      ~11910 callsAsync fiber recursion limit: ~1482 calls

Threads have roughly 8x more stack space than Async fibers. This is becausethreads use larger stacks (1MB-8MB) while fibers are designed to be lightweightwith smaller stacks (512KB-1MB). This difference in stack allocation is whythreads can handle deeper recursion before hitting the overflow limit.

With 233 pairs requiring ~1,160 stack frames, we were right at the edge of theAsync fiber limit. But we were still well within the thread limit, which is whyswitching to Thread.new seemed to fix it.

When we tested with 500 pairs (which some of our larger calendars had), evenThread.new failed with a stack overflow. So it wasn't really a fix, it justpushed the problem further down the road.

The GitHub Trail

After we understood the problem, we searched the Rails repository and found theexact history:

1. April 4, 2024 - PR #51492:This PR changed OR nodes from binary to n-ary to handle multiplechildren in a single node.

2. September 24, 2024 -Issue #53031: Someonereported the exact issue we were seeing. Queries with 800+ OR conditions thatworked in Rails 7.1 now crashed in Rails 7.2.

3. September 25, 2024 -PR #53032:fatkodima fixed it with a one-line change inPredicateBuilder#grouping_queries:

Before:

queries = queries.reduce { |result, query| Arel::Nodes::Or.new([result, query]) }

After:

queries = Arel::Nodes::Or.new(queries)

The difference is profound. The old code using reduce created a deeply nestedstructure like Russian nesting dolls:

Or([Or([Or([Query1, Query2]), Query3]), Query4])  # 232 levels with 233 pairs

The new code creates a flat structure:

Or([Query1, Query2, Query3, Query4, ...Query233])  # Just 1 level

With a flat structure, the recursive visitor only traverses one level instead of232, eliminating the stack overflow. The fix is available in Rails 7.2.2+.

Our Solution

We're currently on Rails 7.2.1.1 and upgrading immediately would requireextensive testing. So we implemented a workaround in our code:

def abandoned_events  return @_abandoned_events if defined?(@_abandoned_events)  pairs = event_params.map { |e| [e[:url], e[:recurrence_id]] }  return if pairs.empty?  events_in_time_range = calendar.events    .where("start_time BETWEEN ? AND ?", start_time, end_time)  # Get candidate events using simple IN clauses  urls = pairs.map(&:first).uniq  recurrence_ids = pairs.map(&:last).uniq  candidate_events = events_in_time_range    .where(url: urls)    .where(recurrence_id: recurrence_ids)  # Filter to exact pairs in memory  pairs_set = pairs.to_set  ids_to_exclude = candidate_events.select { |event|    pairs_set.include?([event.url, event.recurrence_id])  }.map(&:id)  @_abandoned_events = if ids_to_exclude.empty?    events_in_time_range  else    events_in_time_range.where.not(id: ids_to_exclude)  endend

This approach:

• Uses simple IN clauses instead of complex OR conditions
• Filters the exact pairs in memory (fast with a Set)
• Excludes by ID, which is a flat list
• Never builds deeply nested Arel nodes

The trade-off is we run 2 queries instead of 1, but the queries are simpler andmore efficient. The in-memory filtering is negligible since we're alreadyconstrained by time range.

Measuring and Verifying

To prove our hypothesis, we wrote verification scripts that patched Railsinternals to measure what was actually happening.

Counting Arel Visitor Calls

We created a script that patched visit_Arel_Nodes_Or to count how many timesit was called and how deep the recursion went:

module Arel  module Visitors    class ToSql      alias_method :original_visit_Arel_Nodes_Or, :visit_Arel_Nodes_Or      def visit_Arel_Nodes_Or(o, collector)        @or_call_count ||= 0        @or_call_count += 1        @max_depth ||= 0        @current_depth ||= 0        @current_depth += 1        @max_depth = [@max_depth, @current_depth].max        result = original_visit_Arel_Nodes_Or(o, collector)        @current_depth -= 1        result      end    end  endend

Results with 50 test pairs:

• Rails 7.1.5.2: visit_Arel_Nodes_Or was called 1 time and max recursiondepth was 0.
• Rails 7.2.1.1: visit_Arel_Nodes_Or was called 49 times and max recursiondepth was 100.

With 233 pairs in production:

• Rails 7.1.5.2: Still called 1 time, depth 0 (iterative approach)
• Rails 7.2.1.1: Called 232 times, depth ~466 (exceeds Async fiber limit)

Testing the Actual Breaking Points

We ran the problematic query with different pair counts to find exactly where itbreaks:

# Test with AsyncAsync do  pairs = calendar.events.limit(count).pluck(:url, :recurrence_id)  calendar.events.where.not([:url, :recurrence_id] => pairs).countend.wait# Test with ThreadThread.new do  pairs = calendar.events.limit(count).pluck(:url, :recurrence_id)  calendar.events.where.not([:url, :recurrence_id] => pairs).countend.join

Results:

• Async Fiber: Breaks at approximately 233 pairs
• Thread.new: Breaks at approximately 500 pairs

This confirmed that Thread.new wasn't a real solution - it just had moreheadroom before hitting the same problem.

For now, we're sticking with our workaround. It works reliably, performs well,and doesn't require upgrading Rails immediately. When we do upgrade to Rails7.2.2+, we can revert to the original clean syntax, knowing that the Rails teamhas fixed the underlying issue.

If you're on Rails 7.2.0 through 7.2.1.x and use composite key queries withlarge datasets, watch out for this issue. The fix is in Rails 7.2.2+, or you canwork around it like we did.

References

• PR #51492 - Bug introduced
• Issue #53031 - Bug reported
• PR #53032 - Bug fixed

Marking an association as deprecated

Simply pass the deprecated: true option when declaring an association.

class User < ApplicationRecord  has_many :meetings, deprecated: trueend

Now, every time the meeting association is invoked, well get a deprecationwarning in our logs.

> user.meetingsThe association User#meetings is deprecated, the method meetings was invoked ((calendar):4:in '<main>')> User.includes(:meetings).where(id: 1)The association User#meetings is deprecated, referenced in query to preload records ()> User.joins(:meetings).where(id: 1)The association User#meetings is deprecated, referenced in query to join its table ()

Working with different association types

We can deprecate any association type.

class Order < ApplicationRecord  # has_many  has_many :line_items, deprecated: true  # belongs_to  belongs_to :customer, deprecated: true  # has_one  has_one :profile, deprecated: true  # has_many through  has_many :archived_comments, through: :comments, deprecated: trueend

Reporting modes and backtrace support

This feature supports three deprecation modes:

• :warn (default) Logs a warning to our ActiveRecord logger.
• :raise Raises an exception when the deprecated association is used.
• :notify Emits an Active Support notification event with the keydeprecated_association.active_record. This can be used to send notificationsto external services like Honeybadger etc. We can check the details about itspayloadhere.

Backtraces are disabled by default. If :backtrace is true, :warn mode willinclude a clean backtrace in the message, and :notify mode will have abacktrace key in the payload. Exceptions raised via :raise mode will alwayshave a clean stack trace.

We can change the global default mode in an initializer.

# config/initializers/deprecated_associations.rbActiveRecord.deprecated_associations_options = {  mode: :warn,      # :warn | :raise | :notify  backtrace: true   # whether to include a cleaned backtrace}

It can also be set at an environment level.

# config/environments/development.rbRails.application.configure do  config.active_record.deprecated_associations_options = { mode: :raise, backtrace: true }end# config/environments/production.rbRails.application.configure do  config.active_record.deprecated_associations_options = { mode: :warn, backtrace: true }end

Why deprecate rather than remove?

In large applications, its often hard to guarantee complete test coverage. Someassociation usages may only surface in production.

Deprecating an association first, lets us:

• Identify every code path (tests, console, background jobs) that relies on it.
• Gradually refactor those references before removal.
• Ensure confidence that deleting the association wont break anythingunexpectedly.

This new feature in Rails provides a clean and intuitive way to phase out ourassociations with deprecation warnings, making it easier to maintain andrefactor large codebases.

This feature was merged recently,and will be released in the next Rails minor/patch version.

If you prefer watching a video to learn about Active Job Continuations, then wemade a video for you.

<iframewidth="560"height="315"src="https://www.youtube.com/embed/r4uuQh1Zog0"title="YouTube video player"frameborder="0"allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"allowfullscreen

</iframe>

In short, this feature allows you to configure your jobs in such a manner thatthe job can be interrupted and next time when the job starts, it'll start from aparticular point so that the work done so far is not totally wasted.

This work is highly inspired by Shopify'sjob-iteration gem.

Queueing Systems

In web applications, not every task needs to be processed immediately. When youupload a large video file, send a bulk email campaign, or generate a complexreport, these time-consuming operations are often handled in the background.This is where queueing systems like Sidekiq orSolid Queue comes into play.

Queueing theory helps us understand how these systems behave under differentconditions - from quiet periods to peak load times.

Let's understand the fundamentals of queueing theory.

Basic Terminology in Queueing Theory

1. Unit of Work: This is the individual item needing service - a job.

2. Server: This is one "unit of parallel processing capacity." In queuingtheory, this doesn't necessarily mean a physical server. It refers to theability to process one unit of work at a time. For JRuby or TruffleRuby, eachthread can be considered a separate "server" since they can execute inparallel. For CRuby/MRI, because of the GVL, the concept of a "server" isdifferent. We'll discuss it later.

3. Queue Discipline: This is the rule determining which unit of work isselected next from the queue. For Sidekiq and Solid Queue, it is FCFS(FirstCome First Serve). If there are multiple queues, which job is selecteddepends on the priority of the queue.

4. Service Time: The actual time it takes to process a unit of work (howlong a job takes to execute).

5. Latency/Wait Time: How long jobs spend waiting in the queue before beingprocessed.

6. Total Time: The sum of service time and wait time. It's the completeduration from when a job is enqueued until it finishes executing.

Little's law

Little's law is a theorem in queuing theory that states that the average numberof jobs in a system is equal to the average arrival rate of new jobs multipliedby the average time a job spends in the system.

L = W

L = Average number of jobs in the system
= Average arrival rate of new jobs
W = Average time a job spends in the system

For example, if jobs arrive at a rate of 10 per minute (), and each job takes30 seconds (W) to complete:

Average number of jobs in system = 10 jobs/minute * 0.5 minutes = 5 jobs

This helps us understand the current state of our system and that there are 5jobs in the system on average at any given moment.

L is also called offered traffic.

Note: Little's Law assumes the arrival rate is consistent over time.

Managing Utilization

Utilization measures how busy our processing capacity is.

Mathematically, it is the ratio of how much processing capacity we're using tothe processing capacity we have.

utilization = (average number of jobs in the system/capacity to handle jobs) * 100

In other words, it could be written as follows.

utilization = (offered_traffic / parallelism) * 100

For example, if we are using Sidekiq to manage our background jobs then in asingle-threaded case, parallelism is equal to the number of Sidekiq processes.

Let's look at a practical case with numbers:

• We have 30 jobs arriving every minute
• It takes 0.5 minutes to process a job
• We have 20 Sidekiq processes

In this case, the utilization will be:

utilization = ((30 jobs/minute * 0.5 minutes) / 20 processes) * 100 = 75%

High utilization is bad for performance

Let's assume we maintain 100% utilization in our system. It means that if, onaverage, we get 30 jobs per minute, then we have just enough capacity to handle30 jobs per minute.

One day, we started getting 45 jobs per minute. Since utilization is at 100%,there is no extra room to accommodate the additional load. This leads to higherlatency.

Hence, having a high utilization rate may result in low performance, as it canlead to higher latency for specific jobs.

The knee curve

Mathematically, it would seem that only when the utilization rate hits 100%should the latency spike up. However, in the real world, it has been found thatlatency begins to increase dramatically when utilization reaches around 70-75%.

If we draw a graph between utilization and performance then the graph would looksomething like this.

The point at which the curve bends sharply upwards is called "Knee" in theperformance curve. At this point, the exponential effects predicted by queueingtheory become pronounced, causing the queue latency to climb up quickly.

Running any system consistently above 70-75% utilization significantly increasesthe risk of spiking the latency, as jobs spend more and more time waiting.

This would directly impact the customer experience, as it could result in delaysin sending emails or making calls to Twilio to send SMS messages, etc.

Tracking this latency will be covered in the upcoming blogs. The tracking ofmetrics depends on the queueing backend used (Sidekiq or Solid Queue).

Concurrency and theoretical parallelism

In Sidekiq, a process is the primary unit of parallelism. However, concurrency(threads per process) significantly impacts a process's effective throughput.Because of GVL we need to take into account how long the job is waiting for I/O.

The more time a job spends waiting on external resources (like databases orAPIs) rather than executing Ruby code, the more other threads within the sameprocess can run Ruby code while the first thread waits.

We learned about Amdahl's law inPart 3of this series.

Where:

p is the portion that can be parallelized (the I/O percentage)

n is the number of threads (concurrency)

Speedup is equivalent to theoretical parallelism in this context. In queueingtheory, parallelism refers to how many units of work can be processedsimultaneously. When we calculate speedup using Amdahl's Law, we're essentiallydetermining how much faster a multi-threaded system can handle work compared toa single-threaded system.

Let's assume that a system has an I/O of 50% and a concurrency of 10. ThenSpeedup will be:

Speedup = 1 / ((1 - 0.5) + 0.5 / 10) = 1 / 0.55 = 1.82  2

This means one Sidekiq process with 10 threads will handle jobs twice as fast asSidekiq with a single process with a single thread.

Let's recap what we are saying here. We are assuming that the system has I/O of50%. System is using a single Sidekiq process with 10 threads(concurrency). Thenbecause of 10 threads the system has a speed gain of 2x compared to systemhaving just a single thread. In other words, just because we have 10 threadsrunning we are not going to gain 10X performance improvement. What those 10threads is getting us is what is called "theoretical parallelism".

Similarly, for other values of I/O and Concurrency, we can get the theoreticalparallelism.

Let's go over one more time. In the last example, what we are stating is that ifa system has 95% I/O and if the system has 64 threads running, then that willgive 16x performance improvement over the same system running on a singlethread.

Here is the graph for this data.

As shown in the graph, a Sidekiq process with 16 threads handling jobs that are75% I/O-bound achieves a theoretical parallelism of approximately 3. In otherwords, 3x performance improvement is there over a single-threaded system.

Calculating the number of processes required

At the beginning of this article, we discussed "Little's law" and we discussedthat L is also called "offered traffic," which depicts the "average number ofjobs in the system".

If "offered traffic" is 5, then it means we have 5 units of work arriving onaverage that requires processing simultaneously.

We just learned that if the utilization is greater than 75% then it can causeproblems as there is a risk of latency to spike.

For queues with low latency requirements(eg. urgent), we need to target alower utilization rate. Let's say we want utilization to be around 50% to be onthe safe side.

Now we know the utilization rate that we need to target and we know the "offeredtraffic". So now we can calculate the "parallelism".

utilization = offered_traffic / parallelism=> 0.50 = 5 / parallelism=> parallelism = 5 / 0.50 = 10

This means we need a theoretical parallelism of 10 to ensure that theutilization is 50% at max.

Let's assume the jobs in this queue have an average of 50% I/O. Based on theabove mentioned graph, we can see that if the concurrency is 10 then we getparallelism of 2. However, increasing the concurrency doesn't increase theparallelism. It means if we want 10 parallelism, then we can't just switch toconcurrency of 50. Even the concurrency of 50 (or 50 threads) will only yield aparallelism of 2.

So we have no choice but to add more processes. Since one process with 10concurrency is yielding a parallelism of 2 we need to add 5 processes to get 10parallelism.

To get the I/O wait percentage, we can make use of perfm.Hereis the documentation on how it can be done.

Total number of Sidekiq processes required = 10 / 2 = 5

Here we're talking about Sidekiq free version, where we'll only be able to run asingle process per dyno. If we're using Sidekiq Pro, we can run multipleprocesses per dyno via Sidekiq Swarm.

We can provision 5 dynos for the urgent queue. But we should always have a queuetime based autoscaler like Judoscale enabled to handle spikes.

Sources of Saturation

We discussed earlier that, in the context of queueing theory, the saturationpoint is typically reached at around 70-75% utilization. This is from the pointof view of further gains by adding more threads.

However saturation can occur in other parts of the system.

CPU

The servers running your Sidekiq processes have finite CPU and memory. While CPUusage is a metric we can track for Sidekiq, it's generally not the only one weneed to focus on for scaling decisions.

CPU utilization can be misleading. If our jobs spend most of their time doingI/O (like making API calls or database queries), in which case CPU usage will bevery low, even when our Sidekiq system is at capacity.

Memory

Memory utilization impacts performance very differently from CPU utilization.Memory utilization generally exhibits minimal changes in latency or throughputfrom 0% to 100% utilization. However, after 100% utilization, things will startto deteriorate significantly. The system will start using the swap memory, whichcan be very slow and thereby increase the job service times.

Redis

Another place where saturation can occur is in our datastore i.e Redis in caseof Sidekiq. We have to make sure that we provision a separate Redis instance forSidekiq and also make sure to set the eviction policy to noeviction. Thisensures that Redis will reject new data when the memory limit is reached,resulting in an explicit failure rather than silently dropping important jobs.

This was Part 6 of our blog series onscaling Rails applications. If any part of theblog is not clear to you then please write to us atLinkedIn,Twitter orBigBinary website.

Database Connection Pooling

When a Rails application needs to interact with a database, it establishes aconnection, which is a dedicated communication channel between the applicationand the database server. When a new request comes to Rails, the operation can behandled like this.

1. Create a connection
2. Do database operation
3. Close the connection

When the next request comes, then repeat the above process.

Creating new database connections is an expensive operation - it takes time toestablish the connection, authenticate, and set up the communication channel. Itmeans that every single time a request comes, we are spending time in setting upthe connection.

Wouldn't it be better to store the established connection to somewhere and whena new request comes, then get the pre-established connection from this pool.This expedites the process since we don't need to create a connection and closea connection in every single request.

The new process might look like this.

1. Create a connection
2. Do database operation
3. Put the connection in a pool

Now, when a new request comes, the operation will look like this.

1. Get the connection from the pool
2. Do database operation
3. Return the connection to the pool

Database connection pooling is a performance optimization technique thatmaintains a set of reusable database connections.

Active Record Connection Pool Implementation

Active Record manages a pool of database connections for each web and backgroundprocess. Each process will have a connection pool of its own, which means aRails application running with multiple processes (like Puma processes, Sidekiqprocesses) will have multiple independent connection pools. The pool is a set ofdatabase connections that are shared among threads in the same process.

Note that the pooling happens at the process level. A thread from process Acan't get a connection from process B.

When a connection is needed, the thread checks out a connection from the pool,perform operations, and then returns the connection to the pool. This is done ata query level now. For each individual query, a connection is leased, used, andthen returned back to the pool.

Pre Rails 7.2, the connection used to be leased and held till the end of therequest if it is a web request and till the job is done if it is a backgroundjob. This was a problem for applications that spent a lot of time doing I/O. Thethread will hog the connection for the entire duration of the I/O operationlimiting the number of queries that can be executed concurrently. To facilitatethis change and make query caching work, the query cache has beenupdated to be owned by the pool.

This means that the query cache is now shared across all connections in thepool. Previously, each connection had its own query cache. As the whole requestused the same connection, this was fine. But now, as the connection is leasedfor each query, the query cache needs to be shared across all connections in thepool.

Connection Pool Configuration Options

Active Record's connection pool behavior can be customized through severalconfiguration options in the database.yml file:

• pool:Sets the maximum number of connections the pool will maintain. The default istied to RAILS_MAX_THREADS, but you can set it to any value. There is a smallproblem when you set it to RAILS_MAX_THREADS which we'll discuss later.
• checkout timeout:Determines how long a thread will wait to get a connection before timing out.The default is 5 seconds. If all connections are in use and a thread waitslonger than this value, an ActiveRecord::ConnectionTimeoutError exceptionwill be raised.
• idle timeout:Specifies how long a connection can remain idle before it's removed from thepool. The default is 300 seconds. This helps reclaim resources fromconnections that aren't being used.
• reaping frequency:Controls how often the Reaper(which we'll discuss shortly) process runs toremove dead or idle connections. The default is 60 seconds.

Active Record Connection Pool Reaper

Database connections can sometimes become "dead" due to issues like databaserestarts, network problems etc. Active Record provides Reaper to handle this.

The Reaper periodically checks connections in the pool and removes deadconnections as well as idle connections that have been checked out for a longtime.

It acts somewhat like a garbage collector for database connections. The Reaperuses the idle_timeout setting to determine how long a connection can remainidle before being removed, tracking idle time based on when connections werelast used.

There is another configuration option called reaping_frequency that controlshow often the Reaper runs to remove dead or idle connections from the pool. Bydefault, this is set to 60 seconds. It means the Reaper will wake up once everyminute to perform its maintenance tasks.

If your application is spiky and receives a lot of traffic in surges, then setthe reaping frequency and idle timeout to a lower value. This will ensure thatthe reaper runs more frequently and removes idle connections more quickly,helping to keep the connection pool healthy and responsive.

Why are idle connections bad?

Idle database connections can significantly impact database performance forseveral interconnected reasons:

Memory Consumption: Each database connection, even when idle, maintains itsown memory allocation. The database must reserve memory for session state,buffers, user context, and transaction workspace. This memory remains allocatedeven when the connection isn't doing any work. For example, if each connectionuses 10 MB of memory, 100 idle connections would unnecessarily consume 1 GB ofyour database's memory that could otherwise be used for active queries, caching,or other productive work.

CPU overhead: While "idle" suggests no activity, the database still performsregular maintenance work for each connection. It must monitor connection healthvia keepalive checks, manage process tables etc.

The crucial issue is that the overhead of having idle connections scalesnon-linearly. As we add more idle connections, the database spends an increasingproportion of its CPU time just managing these connections rather thanprocessing actual queries. Thankfully, the reaper handles this for us.

How many database connections will the web and background processes utilize at maximum?

As we learned, the connection pool is managed at the process level. Each Railsprocess maintains its own pool.

1. In web processes (Puma):

Each Puma process is a separate process with its own connection pool. In aprocess, each thread can check out one connection. Therefore, maximumconnections needed per process equals max_threads setting in Puma.

2. In background processes (Sidekiq):

Sidekiq runs as a separate process with its own connection pool. The Sidekiqconcurrency setting determines the number of threads and therefore the maximumconnections needed equals the concurrency value.

Note: If you're using Sidekiq swarm and running multiple Sidekiq processes,then take that it into account.

We can calculate the total potential connections for a typical application asshown below.

Web connections = Number of web dynos * Number of Puma processes                  * `max_threads` valueBackground connections = Number of worker dynos * Number of Sidekiq processes                         * Threads per processTotal number of connections = Web connections + Background connections

The key thing to note here is that the database needs to support at least thesemany simultaneous connections.

Note that if preboot is enabled, then the maximum number of connections will bedouble the above value. This is because during the release phase, there is asmall window in which both the old dynos and new dynos are running.

In Rails 7,load_asyncwas introduced which allows us to run database queries asynchronously in abackground thread. When load_async is in used, the calculation for maximumnumber of connections needed changes a bit. First, let's understand howload_async works.

How load_async works

load_async allows Rails to execute database queries asynchronously inbackground threads. Unlike regular ActiveRecord queries which are lazily loaded,load_async queries are always executed immediately in background threads andjoined to the main thread when results are needed.

The async executor is configured through theconfig.active_record.async_query_executor setting. There are three possibleconfigurations:

1. nil (default): Async queries are disabled, and load_async will executequeries synchronously.
2. :global_thread_pool: Uses a single thread pool for all databaseconnections.
3. :multi_thread_pool: Uses separate thread pools for each databaseconnection.

Rails provides a configuration option namedglobal_executor_concurrency(default: 4) that controls how many concurrent async queries can run perprocess. So, the maximum number of connections per process when load_async isused.

Maximum connections per process = Process level concurrency                                  + global_executor_concurrency + 1

Here Process level concurrency means max_threads for the Puma process andsidekiq_conurrency for the Sidekiq process.

The "+1" accounts for the main control thread, which may occasionally need aconnection (e.g., during model introspection at class load time).

There is a nicecalculatorcreated by the folks at Judoscale which can be used tocalculate the maximum number of connections needed for your application.

Setting Database Pool Size Configuration

Our database.yml file has the following line.

pool: <%%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>

We know that a thread doesn't take more than one DB connection. So the maximumnumber of connections needed per pool is equal to the total number of threads.So the above configuration looks fine.

However this doesn't take into account whether we use load_async or not. If weuse load_async, then the number of connections needed per process will beRAILS_MAX_THREADS + global_executor_concurrency + 1.

Do we really need to go into this much detail to determine the pool size? Turnsout there is a much easier answer.

Almost all database hosting providers mention the maximum number of connectionsallowed in their plan. We can just set the pool config to the maximum number ofconnections supported by our database plan. Let us say we have a Standard-0database on Heroku. It supports up to 120 connections. So we can set the poolconfig to 120.

pool: 120

We can do this because the database connections are lazily initialized in thepool. The application doesn't create more database connections than it needs. Sowe needn't be conservative here.

The only thing we need to ensure is that the maximum connection utilizationdoesn't exceed the database plan limit. If that happens, then we have anothersolution - PgBouncer.

PgBouncer

PgBouncer is a lightweight connection pooler for PostgreSQL. It sits between ourapplication(s) and our PostgreSQL database and manages a pool of databaseconnections.

While both PgBouncer and Active Record provide connection pooling, they operateat different levels and serve different purposes.

Active Record connection pool operates within a single Ruby process and managesconnections for threads within the process, whereas PgBouncer is an externalconnection pooler that sits between the application and the database and managesconnections across all the application processes.

The dreaded ActiveRecord::ConnectionTimeoutError

This error comes up when a thread waits more than checkout_timeout seconds toacquire a connection. This usually happens when the pool size is set to avalue less than the concurrency.

For example, lets say we have set the Sidekiq concurrency to 10 and pool sizeto 5. If we have more than 5 threads wanting a connection at any point of time,the threads will have to wait.

What's the solution? As we discussed earlier setting the pool to a really highvalue should fix the error in most cases.

Even after setting the config correctly, ActiveRecord::ConnectionTimeoutErrorcan still happen and it could be puzzling. Let is discuss a few scenarios wherethis can happen.

Custom code spinning up new threads and taking up connections

class SomeService  def process    threads = []    5.times do |index|      threads << Thread.new do        ActiveRecord::Base.connection.execute("select pg_sleep(5);")      end    end    threads.each(&:join)  endend

Here 5 threads are spun up. Note that these threads also take up connectionsfrom the same pool allotted to the process.

Active Storage proxy mode

Even if our application code is not spinning up new threads, Rails itself cansometimes spin up additional threads. For example Active Storage configured inproxy mode.

Active Storages proxy controllers1,2generate responses as streams, which require dedicated threads for processing.

This means that when serving an Active Storage file through one of these proxycontrollers, Rails actually utilizes two separate threads - one for the mainrequest and another for the streaming process. Each of these threads requiresits own separate database connection from the ActiveRecord connection pool.

Rack timeouts

rack-timeout is commonly used acrossRails applications to automatically terminate long-running requests. While ithelps prevent server resources from being tied up by slow requests, it can alsocause a few issues.

Rack timeout uses Ruby'sThread#raise API to terminaterequests that exceed the configured timeout. When a timeout occurs, rack-timeoutraises a Rack::Timeout::RequestTimeoutException from another thread. If thisexception is raised while a thread is in the middle of database operations, itcan prevent proper cleanup of database connections.

Tracking down ActiveRecord::ConnectionTimeoutErrors

If we still frequently see ActiveRecord::ConnectionTimeoutError exceptions inour application, we can get additional context by logging the connection poolinfo to our error monitoring service. This can help identify which all threadswere holding onto the connections when the error occurred.

config.before_notify do |notice|  if notice.error_class == "ActiveRecord::ConnectionTimeoutError"    notice.context = { connection_pool_info: detailed_connection_pool_info }  endenddef detailed_connection_pool_info  connection_info = {}  ActiveRecord::Base.connection_pool.connections.each_with_index do |conn, index|    connection_info["connection_#{index + 1}"] = conn.owner ? conn.owner.inspect : "[UNUSED]"  end  connection_info["current_thread"] = Thread.current.inspect  connection_infoend

<thread_obj>.inspect gives us the name, id and status of the thread. Forexample, if one entry in the hash looks like#<Thread:0x00006a42eca73ba0@puma srv tp 002 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 sleep_forever>then it means that the connection is taken up by a Puma thread.

Monitoring Active Record Connection Pool Stats

If we want to monitor Active Record Connection Pool stats, then periodically weneed to send the stats to a service provider which can display the datagraphically. For periodically checking the stat, we are usingrufus-scheduler gem. Forcollecting the data and showing the data we are using NewRelic but you can useany APM of your choice. We have configured to send the pool stat every 15seconds.

Here is thegist which collects and sends data.

This was Part 5 of our blog series onscaling Rails applications. If any part of theblog is not clear to you then please write to us atLinkedIn,Twitter orBigBinary website.

Inpart 1we saw how to find ideal number of processes for our Rails application.

Inpart 3,we learned about Amdahl's law, which helps us find the ideal number of threadstheoretically.

In this blog, we'll run a bunch of tests on our real production application tosee what the actual number of threads should be for each process.

Inpart 1we discussed the presence of GVL and the concept of thread switching. Based onthe GVL's interaction, a thread can be in one of these three states.

1. Running: The thread has the GVL and is executing Ruby code.
2. Idle: The thread doesn't want the GVL because it is performing I/Ooperations.
3. Stalled: The thread wants the GVL and is waiting for it in the GVL waitqueue.

Based on the above diagram, we can approximately equate idle time toI/O time.

GVL instrumentation using perfm

Thanks to Jean Boussier's work on theGVL instrumentation API and JohnHawthorn's work on gvl_timing, we cannow measure the time a thread spends in each of these states for apps running onRuby 3.2 or higher.

Using the great work done by these folks, we have createdperfm, to help us figure out the idealnumber of Puma threads based on the application's workload.

Perfm inserts a Rack middleware to our Rails application. This middlewareinstruments the GVL, collects the required metrics and stores them in a table.It also has a Perfm::GvlMetricsAnalyzer class which can be used to generate areport on the data collected.

Using perfm to measure the application's I/O percentage

To use perfm, we need to add the following line to our Gemfile.

gem 'perfm'

We'll run bin/rails generate perfm:install. This will generate the migrationto create perfm_gvl_metrics which will be used to store request-level metrics.

Now we'll create an initializer config/initializers/perfm.rb.

Perfm.configure do |config|  config.enabled = true  config.monitor_gvl = true  config.storage = :localendPerfm.setup!

After deploying the code to production, we need to collect around 20K requestsas that will give us a fair number of data points to analyze. The GVL monitoringcan be disabled after that by setting config.monitor_gvl to false so thatthe table doesn't keep growing.

After collecting the request data, now it's time to analyze it.

Run the following code in the Rails console.

irb(main):001* gvl_metrics_analyzer = Perfm::GvlMetricsAnalyzer.new(irb(main):002*   start_time: 2.days.ago, # configure thisirb(main):003*   end_time: Time.currentirb(main):004> )irb(main):005>irb(main):006> results = gvl_metrics_analyzer.analyzeirb(main):007> io_percentage = results[:summary][:total_io_percentage]=> 45.09

This will give us the percentage of time spent doing I/O. We ran it in ourNeetoCal production application and we got a value of45%.

As we discussed inpart 3,Amdahl's law gives us a theoretical maximum speedup based on the parallelizableportion of our workload. The formula is given below.

Where:

• p is the portion that can be parallelized (in our case, it's 0.45)
• N is the number of threads
• (1 - p) is the portion that must run sequentially (in our case, it's 0.55)

Let's calculate the theoretical speedup for different numbers of threads with p= 0.45:

We can see that after 4 threads, the percentage improvement drops below 5%. Thismeans that 4 is a reasonable value for max_threads. We can set the value ofRAILS_MAX_THREADS to 4.

Looking at the table, adding a 5th thread would only give us a 3% performanceimprovement, which may not justify the additional memory usage and potential GVLcontention.

We have also created a small application to help visualize and find the idealnumber of threads when the I/O percentage is provided as input.Here is the link to theapp.

Validate thread count using stall time

This value of 4 we got theoretically by using Amdahl's law. Now it's time toput this law to test. Let's see in the real world if the value of 4 is thecorrect value or not.

What we need to do is start with RAILS_MAX_THREADS env variable (Pumamax_threads) set to 4 and then check if this value provides minimal GVLcontention. By GVL contention, we mean the amount of time a thread spendswaiting for the GVL i.e the stall time.

If the stall time is high, that means the set thread count is high. We don'twant our request threads to spend their time doing nothing causing latencyspikes.75ms is an acceptable value for stall time. The lesser the better ofcourse.

The average stall time can be found in the perfm analyzer results. As wementioned earlier, we had collected data for NeetoCal.Now let's find the average stall time.

irb(main):001* gvl_metrics_analyzer = Perfm::GvlMetricsAnalyzer.new(irb(main):002*   start_time: 2.days.ago,irb(main):003*   end_time: Time.current,irb(main):004*   puma_max_threads: 4irb(main):005> )irb(main):006> results = gvl_metrics_analyzer.analyzeirb(main):007> avg_stall_ms = results[:summary][:average_stall_ms]=> 110.24

The stall time seems a bit high. Let us decrease the RAILS_MAX_THREADS valueby 1 and collect a few data points(i.e around 20K requests). Now the value ofRAILS_MAX_THREADS will be 3. This process has to be repeated until we findthe value for which the average stall time is less than 75ms.

irb(main):001* gvl_metrics_analyzer = Perfm::GvlMetricsAnalyzer.new(irb(main):002*   start_time: 2.days.ago,irb(main):003*   end_time: Time.current,irb(main):004*   puma_max_threads: 3irb(main):005> )irb(main):006> results = gvl_metrics_analyzer.analyzeirb(main):007> avg_stall_ms = results[:summary][:average_stall_ms]=> 79.38

Now the output is closer to 75 ms.

Hence we can finalize on the value 3 as the value for RAILS_MAX_THREADS. If wedecrease the value again by one i.e set it to 2, the stall time will decreasebut we're limiting the concurrency of our application. It is a trade-off.

Remember that our goal is to maximize concurrency while minimizing GVLcontention. But if our app spends a lot of time doing I/O - for instance, if wehave a proxy application that makes a lot of external API calls directly fromthe controller, then we can switch the app server toFalcon. Falcon is tailor-made for such usecases.

Broadly speaking, one should take care of the following items to ensure that thetime spent by the request doing I/O is minimal.

• Remove N+1 queries
• Remove long-running queries
• Move inline third party API calls to background job processor
• Move heavy computational stuff to background job processor

For a finely optimized Rails application, the max_threads value will bearound 3. That's why the default value of max_threads for Rails applicationsis 3 now. This has been decided after a lot of discussionhere. We recommend you read thewhole discussion. It is very interesting.

This was Part 4 of our blog series onscaling Rails applications. If any part of theblog is not clear to you then please write to us atLinkedIn,Twitter orBigBinary website.

We have only two parameters to work with if we want to fine-tune our Pumaconfiguration.

1. The number of processes.
2. The number of threads each process can have.

Inpart 1of Scaling Rails series,we saw what the number of processes should be. Now let's look at what the numberof threads in each process should be.

Amdahl's law

Each application has a few things which must be performed in "serial order" anda few things which can be "parallelized". If we draw a diagram, then this iswhat it will look like.

Let's say that T1' and T2' are the enhanced times. These are the times theapplication would take after the enhancement has been applied. In this case theenhancement will come in the form of increasing the threads in a process.

T1' will be same as T1 since it's the serial part. T2' will be lower thanT2 since we will parallelize some of the code. After the parallelization isdone, the enhanced version would look something like this.

It's clear that the serial part (T1) will limit how much speedup we can get nomatter how much we parallelize T2.

Computer scientist Gene Amdahl cameup with Amdahl's law which givesthe mathematical value for the overall speedup that can be achieved.

I made a video explaining how this formula came about.

<iframewidth="966"height="604"src="https://www.youtube.com/embed/2hYs2X6Fb1M?si=f-P_-pcwnotnyUKT"title="Amdahl's Law"frameborder="0"allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"allowfullscreen

</iframe>

Amdahl's law states that the theoretical speedup gained from parallelization isdirectly determined by the fraction of sequential code in the program.

Now, let's see how we can use Amdahl's law to determine the ideal number ofthreads.

The parallelizable portion in this case is the portion of the application thatspends time doing I/O. The non-parallelizable portion is the time spent by theapplication executing Ruby code. Remember that because of GVL, within a process,only one thread has access to CPU at any point of time.

Now we need to know what percentage of time our app spends doing I/O. This willbe the value p as per the video.

Later in this series, we'll show you how to calculate what percentage of thetime the Rails application is spending doing I/O. For this blog let's assumethat our application spends 37% of the time doing I/O i.e the value of p is0.37.

Let's calculate how much speedup we will get if we use one thread(n is 1). Nowlet's change n to 2 and get the speedup value. Similarly, we bump up n all theway to 15 and we record the speedup.

Now let's draw a graph between the overall speedup and the number of threads.

From the graph, it can be seen that the speedup increases as the number ofthreads increases, but the rate of increase diminishes as more threads areadded. This is because the serial portion remains constant and is unaffected bythe increase in threads.

By examining the graph we can observe that the speedup gain from increasingthreads seem significant up to 4 threads, after which the incremental gain inspeedup starts to plateau.

Remember that these are theoretical maximums based on Amdahl's law. In practice,we need to use fewer threads as adding more threads can cause an increase inmemory usage and GVL contention, thereby causing latency spikes.

It's obvious that if we add more threads then more requests can be handled byPuma concurrently. What it means is that requests will be waiting for lessertime at the load balancer layer as there are more Puma threads waiting to pickup the request for processing. But in part 1, we saw that just because we havemore threads, it doesn't mean things will move faster. More threads might causeother threads to wait for the GVL.

There is no point in accepting requests if our web server can't respond to itpromptly. Whereas, if the max_threads value is set to a lower value, requestswill queue up at the Load Balancer layer which is better than overwhelming theapplication server.

If more and more requests are waiting at the load balancer level, then therequest queue time will shoot up. The right way to solve this problem is to addmore Puma processes. It is advised to increase the capacity of the Puma serverby adding more processes rather than increasing the number of threads.

Hereis a middleware that can be used to track the request queue time. This code istaken fromjudoscale.

Note that Request Queue Time is the time spent waiting before the request ispicked up for processing.

This middleware will only work if the load balancer is adding theHTTP_REQUEST_START header. Heroku automatically adds this header.

Now we need to use this middleware and for that open config/application.rbfile and we need to add the following line.

config.middleware.use RequestQueueTimeMiddleware

This was Part 3 of our blog series onscaling Rails applications. If any part of theblog is not clear to you then please write to us atLinkedIn,Twitter orBigBinary website.

Let's start from the basics. Let's see how a standard web application mostlybehaves.

Web applications and CPU usage

Code in a web application typically works like this.

• Do some data manipulation.
• Make a few database calls.
• Do more calculations.
• Make some network calls.
• Do more calculations.

Visually, it'll look something like this.

CPU work includes operations like view rendering, string manipulation, any kindof business logic processing etc. In short, anything that involves Ruby codeexecution can be considered CPU work. For the rest of the work, like databasecalls, network call etc. CPU is idle. Another way of looking at when the CPU isworking and when it's idle is this picture.

When a program is using CPU, then that portion of the code is called CPUbound and when program is not using CPU, then that portion of the code iscalled IO bound.

CPU bound or IO bound

Let us understand what CPU bound truly means. Consider the following pieceof code.

10.times do  Net::HTTP.get(URI.parse("https://bigbinary.com"))end

In the above code, we are hitting the BigBinary website 10 times sequentially.Running the above code takes time because making a network connection is atime-consuming process.

Let's assume the code above takes 10 seconds to finish. We want the code to runfaster. So we bought a better CPU for the server. Do you think now the code willrun faster?

It will not. That's because the above code is not CPU bound. CPU is not thelimiting factor in this case. This code is I/O bound.

A program is CPU bound if the program will run faster if the CPU werefaster.

A program is I/O bound if the program will run faster if the I/O operationswere faster.

Some of the examples of I/O bound operations are:

• making database calls: reading data from tables, creating new tables etc.
• making network calls: reading data from a website, sending emails etc.
• dealing with file systems: reading files from the file system.

Previously, we saw that our CPU was idle sometimes. Now we know that thetechnical term for that idleness is IO bound let's update the picture.

When a program is I/O bound, the CPU is not doing anything. We don't wantprecious CPU cycles to be wasted. So what can we do so that the CPU is fullyutilized?

So far we have been dealing with only one thread. We can increase the number ofthreads in the process. In this way, whenever the CPU is done executing CPUbound code of one thread and that thread is doing an I/O bound operation, thenthe CPU can switch and handle the work from another thread. This will ensurethat the CPU is efficiently utilized. We will look at how the switching betweenthreads works a bit later in the article.

Concurrency vs Parallelism

Concurrency and parallelism sound similar, and in your daily life, you cansubstitute one for another, and you will be fine. However, from the computerengineering point of view, there is a difference between work happeningconcurrently and work happening in parallel.

Imagine a person who has to respond to 100 emails and 100 Twitter messages. Theperson can reply to an email and then reply to a Twitter message, then do thesame all over again: reply to an email and reply to a Twitter message.

The boss will see the count of pending emails and Twitter messages go down from100 to 99 to 98. The boss might think that the work is happening in "parallel."But that's not true.

Technically, the work is happeningconcurrently. For a system to be parallel, itshould have two or more actions executed simultaneously. In this case, at anygiven moment, the person was either responding to email or responding toTwitter.

Another way to look at it is that Concurrency is about dealing with lots ofthings at the same time. Parallelism is about doing lots of things at thesame time.

If you find it hard to remember which one is which then remember thatconcurrency starts with the word con. Concurrency is the conman. It'spretending to be doing things "in parallel," but it's only doing thingsconcurrently.

Understanding GVL in Ruby

GVL (Global VM Lock) in Ruby is a mechanism that prevents multiple threads fromexecuting Ruby code simultaneously. The GVL acts like a traffic light in aone-lane bridge. Even if multiple cars (threads) want to cross the bridge at thesame time, the traffic light (GVL) allows only one car to pass at a time. Onlywhen one car has made it safely to the other end, the second car is allowed bythe traffic light(GVL) to start.

Ruby's memory management(like garbage collection) and some other parts of Rubyare not thread-safe. Hence, GVL ensures that only one thread runs Ruby code at atime to avoid any data corruption.

When a thread "holds the GVL", it has exclusive access to modify the VMstructures.

It's important to note that GVL is there to protect how Ruby works and managesRuby's internal VM state. GVL is not there to protect our application code. It'sworth repeating. The presence of GVL doesn't mean that we can write our code ina thread unsafe manner and expect Ruby to take care of all threading issues inour code.

Ruby offers tools like Mutex andconcurrent-ruby gem tomanage concurrent code. For example, the followingcode(source) is not threadsafe and the GVL will not protect our code from race conditions.

from = 100_000_000to = 050.times.map do  Thread.new do    while from > 0      from -= 1      to += 1    end  endend.map(&:join)puts "to = #{to}"

When we run this code, we might expect the result to always equal 100,000,000since we're just moving numbers from from to to. However, if we run itmultiple times, we'll get different results.

This happens because multiple threads are trying to modify the same variables(from and to) simultaneously without any synchronization. This is called racecondition and it happens because the operation to += 1 and from -= 1 arenon-atomic at the CPU-level. In simpler terms operation to += 1 can be writtenas three CPU-level operations.

1. Read current value of to.
2. Add 1 to it.
3. Store the result back to to.

To fix this race condition the above code can be re-written using aMutex.

from = 100_000_000to = 0lock = Mutex.new50.times.map do  Thread.new do    while from > 0      lock.synchronize do        if from > 0          from -= 1          to += 1        end      end    end  endend.map(&:join)puts "to = #{to}"

It's worth nothing that Ruby implementations like JRuby and TruffleRuby don'thave a GVL.

GVL dictates how many processes we will need

Let's say that we deploy the production app to AWS's EC2's t2.medium machine.This machine has 2 vCPU as we can see from this chart.

Without going into CPU vs vCPU discussion, let's keep things simple and assumethat the AWS machine has two cores. So we have deployed our code on a machinewith two cores but we have only one process running in production. No worries.We have three threads. So three threads can share two cores. You would thinkthat something like this should be possible.

But it's not possible. Ruby doesn't allow it.

Currently, it's not possible because Thread 1 and Thread 2 belong to the sameprocess. This is because of the Global VM Lock (GVL).

The GVL ensures that only one thread can execute CPU bound code at a time withina single Ruby process. The important thing to note here is that this lock isonly for the CPU bound code and only for the same process.

In the above case, all three threads can do DB operations in parallel. But twothreads of the same process can't be doing CPU operations in parallel.

We can see that "Thread 1" is using Core 1. Core 2 is available but "Thread 2"can't use Core 2. GVL won't allow it.

Again, let's revisit what GVL does. For the CPU bound code GVL will ensure thatonly one thread from a process can access CPU.

So now the question is how do we utilize Core 2. Well, the GVL is applied at aprocess level. Threads of the same process are not allowed to do CPU operationsin parallel. Hence, the solution is to have more processes.

To have two Puma processes we need to set the value of env variableWEB_CONCURRENCY to 2 and reboot Puma.

WEB_CONCURRENCY=2 bundle exec rails s

Now we have two processes. Now both Core 1 and Core 2 are being utilized.

What if the machine has 5 cores. Do we need 5 processes?

Yes. In that case, we will need to have 5 processes to utilize all the cores.

Therefore for achieving maximum utilization, the rule of thumb is that thenumber of processes i.e WEB_CONCURRENCY should be set to the number of coresavailable in the machine.

Thread switching

Now let's see how switching between threads happens in a multi-threadedenvironment. Note that the number of threads is 2 in this case.

As we can see, the CPU switches between Thread 1 and Thread 2 whenever it'sidle. This is great. We don't waste CPU cycles now, as we saw in thesingle-threaded case. But the switching logic is much more nuanced than what isshown in the picture.

Ruby manages multiple threads at two levels: the operating system level and theRuby level. When we create threads in Ruby, they are "native threads" - meaningthey are real threads that the operating system (OS) can see and manage.

All operating systems have a component called the scheduler. In Linux, it'scalled theCompletely Fair Scheduleror CFS. This scheduler decides which thread gets to use the CPU and for howlong. However, Ruby adds its own layer of control through the Global VM Lock(GVL).

In Ruby a thread can execute CPU bound code only if it holds the GVL. The RubyVM makes sure that a thread can hold the GVL for up to 100 milliseconds. Afterthat the thread will be forced to release GVL to another thread if there isanother thread waiting to execute CPU bound code. This ensures that the waitingRuby threads are notstarved.

When a thread is executing CPU-bound code, it will continue until either:

1. It completes its CPU-bound work.
2. It hits an I/O operation (which automatically releases the GVL).
3. It reaches the limit of 100ms.

When a thread starts running, the Ruby VM uses a background timer thread at theVM level that checks every 10ms how long the current Ruby thread has beenrunning. If the thread has been running longer than the thread quantum (100ms bydefault), the Ruby VM takes back the GVL from the active thread and gives it tothe next thread waiting in the queue. When a thread gives up the GVL (eithervoluntarily or is forced to give up), the thread goes to the back of the queue.

The default thread quantum is 100ms and starting from Ruby 3.3, it can beconfigured using the RUBY_THREAD_TIMESLICE environment variable.Here is the link to the discussion.This environment variable allows fine-tuning of thread scheduling behavior - asmaller quantum means more frequent thread switches, while a larger quantummeans fewer switches.

Let's see what happens when we have two threads.

1. T1 completes quantum limit of 100ms and gives up the GVL to T2.
2. T2 completes 50ms of CPU work and voluntarily gives up the GVL to do I/O.
3. T1 completes 75 ms of CPU work and voluntarily gives the GVL to do I/O.
4. Both T1 and T2 are doing I/O and doesn't want the GVL.

It means that Thread 2 would be a lot faster if it had more access to CPU. Tomake CPU instantly available, we can have lesser number of threads CPU has tohandle. But we need to play a balancing game. If the CPU is idle then we arepaying for the processing cost for no reason. If the CPU is extremely busy thenthat means requests will take longer to process.

Thread switching can lead to misleading data

Let's take a look at a simple code given below. This code is taken froma blogby Jean Boussier.

start = Time.nowdatabase_connection.execute("SELECT ...")query_duration = (Time.now - start) * 1000.0puts "Query took: #{query_duration.round(2)}ms"

The code looks simple. If the result is say Query took: 80ms then you wouldthink that the query actually took 80ms. But now we know two things

• Executing database query is an IO operation (IO bound)
• Once the IO bound operation is done then the thread might not immediately gethold of the GVL to execute CPU bound code.

Think about it. What if the query took only 10ms and the rest of the 70msThe thread was waiting for the CPU because of the GVL. The only way to knowwhich portion took how much time is by instrumenting the GVL.

Visualizing the effect of the GVL

To better understand the effect of multiple threads when it comes to Ruby'sperformance, let's do a quick test. We'll start with a cpu_intensive methodthat performs pure arithmetic operations in nested loops, creating a workloadthat is heavily CPU dependent.

Hereis the code.

Running this script produced the following output:

Running demonstrations with GVL tracing...Starting demo with 1 threads doing CPU-bound workTime elapsed: 7.4921 secondsStarting demo with 3 threads doing CPU-bound workTime elapsed: 7.8146 seconds

From the output, we can see that for CPU-bound work, a single thread performedbetter. Why? Let's visualize the result with the help of the traces generated inthe above script using the gvl-tracinggem. The trace files can be visualized usingPerfetto, which provides a timeline view showing howthreads interact with the GVL.

We can see above that in the case of CPU-bound work if we have a single threadthen it's not waiting for GVL. However, if we have three threads then eachthread is waiting for GVL multiple times.

Understanding the advantage of multiple threads in mixed workloads

Now let's look at mixed workloads in single-threaded and multi-threadedenvironment. We'll use a separate script with a mixed_workload method thatcombines CPU-bound work with I/O operations. We use IO.select with blockingbehavior to simulate I/O operations. This creates actual I/O blocking thatreleases the GVL and shows as "waiting" in the GVL trace, accuratelyrepresenting real-world I/O operations like database queries.

Hereis the code for the mixed workload test.

Running this script with 1 thread and 3 threads produced the following output:

Running demonstrations with GVL tracing...Starting demo with 1 thread doing Mixed I/O and CPU workTime elapsed: 9.32 secondsStarting demo with 3 threads doing Mixed I/O and CPU workTime elapsed: 6.1344 seconds

The key advantage of multiple threads in mixed workloads lies in how the GVL ismanaged during I/O operations. When a thread encounters an I/O operation (like adatabase query, network call, or file read), it voluntarily releases the GVL.This is fundamentally different from CPU-bound work, where threads compete forthe GVL and one thread must wait for another to finish or reach the 100msquantum limit.

During I/O operations, the thread is essentially blocked waiting for an externalresource (database, network, disk). While waiting, the thread doesn't need theGVL because it's not executing Ruby code. This creates an opportunity for otherthreads to acquire the GVL and do useful CPU work. The result is that CPU cyclesthat would otherwise be wasted during I/O waits are now being utilizedproductively by other threads.

Let's visualize this with the single-threaded case first:

In the single-threaded case, the threads wait for I/O operations to complete.During these I/O waits, the CPU sits idle. The thread performs some CPU work,then waits for I/O, then does more CPU work, then waits for I/O again. Duringall the I/O wait periods, no productive work is happening. The CPU is availablebut there's no other thread to utilize it.

Now let's look at the multi-threaded case with three threads:

When there are three threads, the situation changes a bit. Threads nowoccasionally spend time waiting for the GVL, but the overall throughput issignificantly better.

When Thread 1 releases the GVL to perform I/O, Thread 2 can immediately acquireit and start executing CPU-bound work. While Thread 2 is working, Thread 1 mightstill be waiting for its I/O operation to complete. Then when Thread 2 releasesthe GVL for its own I/O operation, Thread 3 can acquire it. This creates apipeline effect where threads are constantly handing off the GVL to each otherensuring that the CPU is almost always doing useful work.

The small amount of GVL contention we see in the multi-threaded case (threadswaiting for GVL) is more than compensated for by the elimination of idle CPUtime. Instead of the CPU sitting idle during I/O operations, other threads keepit busy.

This is why Rails applications with typical workloads (lots of database queries,API calls, and other I/O operations) benefit significantly from having multiplethreads.

Why can't we increase the thread count to a really high value?

In the previous section, we saw that increasing the number of threads can helpin utilizing the CPU better. So why can't we increase the number of threads to areally high value? Let us visualize it.

In the hope of increasing performance, let us bump up the number of threads inthe previous code snippet to 20 and see the gvl-tracing result.

As we can see in the above picture, the amount of GVL contention is massivehere. Threads are waiting to get a hold of the GVL. Same will happen inside aPuma process if we increase the number of threads to a very high value. As weknow, each request is handled by a thread. GVL contention therefore, means thatthe requests keep waiting, thereby increasing latency.

What's next

In the coming blogs, we'll see how we can figure out the ideal value formax_threads, both theoretically and empirically, based on our application'sworkload.

This was Part 2 of our blog series onscaling Rails applications. If any part of theblog is not clear to you then please write to us atLinkedIn,Twitter orBigBinary website.

If we dorails newto create a new Railsapplication,Pumawill be the default web server. Let's startby explaining how Puma handles requests.

How does Puma handle requests?

Puma listens to incoming requests on a TCP socket. When a request comes in, thenthat request is queued up in the socket. The request is then picked up by a Pumaprocess. In Puma, a process is a separate OS process that runs an instance ofthe Rails application.

Note that Puma official documentation calls a Puma process a Puma worker. Sincethe term "worker" might confuse people with background workers like Sidekiq orSolidQueue, in this article, we have used the word Puma process at a few placesto remove any ambiguity.

Now, let's look at how a request is processed by Puma step-by-step.

1. All the incoming connections are added to the socket backlog, which is an OSlevel queue that holds pending connections.

2. A separate thread (created by theReactor class) reads theconnection from the socket backlog. As the name suggests, this Reactor classimplements thereactor pattern. The reactorcan manage multiple connections at a time thanks to non-blocking I/O and anevent-driven architecture.

3. Once the incoming request is fully buffered in memory, the request is passedto the thread pool where the request resides in the @todo array.

4. A thread in the thread pool pulls a request from the @todo array andprocesses it. The thread calls the Rack application, which, in our case is aRails application, and generates a response.

5. The response is then sent back to the client via the same connection. Oncethis is complete, the thread is released back to the thread pool to handlethe next item from the @todo array.

Modes in Puma

1. Single Mode: In single mode, only a single Puma process boots and doesnot have any additional child processes. It is suitable only for applicationswith low traffic.

   

2. Cluster Mode: In cluster mode, Puma boots up a master process, whichprepares the application and then invokes thefork() system call tocreate one or more child processes. These processes are the ones that areresponsible for handling requests. The master process monitors and managesthese child processes.

Default Puma configuration in a new Rails application

When we create a new Rails 8 or higher application, the default Pumaconfig/puma.rb will have the following code.

Please note that we are mentioning Rails 8 here because the Puma configurationis different in prior versions of Rails.

threads_count = ENV.fetch("RAILS_MAX_THREADS", 3)threads threads_count, threads_countrails_env = ENV.fetch("RAILS_ENV", "development")environment rails_envcase rails_envwhen "production"  workers_count = Integer(ENV.fetch("WEB_CONCURRENCY", 1))  workers workers_count if workers_count > 1  preload_app!when "development"  worker_timeout 3600end

For a brand new Rails application, the env variables RAILS_MAX_THREADS andWEB_CONCURRENCY won't be set. This means threads_count will be set to 3 andworkers_count will be 1.

Now let's look at the second line from the above mentioned code.

threads threads_count, threads_count

In the above code, threads is a method to which we are passing two arguments.The default value of threads_count is 3. So effectively, we are calling methodthreads like this.

threads(3, 3)

The threads method in Puma takes two arguments: min and max. These argumentsspecify the minimum and maximum number of threads that each Puma process willuse to handle requests. In this case Puma will initialize 3 threads in thethread pool.

Now let's look at the following line from the above mentioned code.

workers workers_count if workers_count > 1

The value of workers_count in this case is 1, so Puma will run in singlemode. As mentioned earlier in Puma a worker is basically a process. It's notbackground job worker.

What we have seen is that if we don't specify RAILS_MAX_THREADS orWEB_CONCURRENCY then, by default, Puma will boot a single process and thatprocess will have three threads. In other words Rails will boot with the abilityto handle 3 requests concurrently.

This is the default value for Puma for Rails booting in development or inproduction mode - a single process with three threads.

Configuring Puma's concurrency and parallelism

When it comes to concurrency and parallelism in Puma, there are two primaryparameters we can configure: the number of threads each process will have andthe number of processes we need.

To figure out the right value for each of these parameters, we need to know howRuby works. Specifically, we need to know how GVL in Ruby works and how itimpacts the performance of Rails applications.

We also need to know what kind of Rails application it is. Is it a CPU intensiveapplication, IO intensive or somewhere in between.

Don't worry in the next blog, we will start from the basics and will discuss allthis and much more.

This was Part 1 of our blog series onscaling Rails applications. If any part of theblog is not clear to you then please write to us atLinkedIn,Twitter orBigBinary website.

Dual-booting is a process that allows you to run your application with differentsets of dependencies, making it easy to switch between them. This approachenables you to quickly test your code with both the current and the newerversion of what you are upgrading, ensuring that everything works smoothlybefore fully committing to the upgrade.

How to dual-boot Ruby?

The dual-boot technique involves maintaining two separate Gemfile.lock fileswithin your Rails project: one for the current version of Ruby and another forthe next version you're upgrading to.

To get started, first create a symbolic link for Gemfile.next with thefollowing command:

ln -s Gemfile Gemfile.next

This command creates Gemfile.next file, which you'll use for the new Rubyversion. This file isn't a separate copy but rather a pointer to your originalGemfile in the Rails project directory.

Next, add the following snippet at the top of your Gemfile:

def next?  File.basename(__FILE__) == "Gemfile.next"end

This code snippet helps determine which Gemfile is in use, whether it's thestandard one or the Gemfile.next for the new Ruby version.

Now, let's utilize the next? method to dynamically set the Ruby version in theGemfile using a conditional operator:

ruby_version = next? ? "3.3.0" : "3.2.2"ruby ruby_version

This code snippet helps determine the Ruby version based on the next? method.If next? returns true, meaning you're operating within Gemfile.next, it setsthe Ruby version to "3.3.0". Otherwise, if the standard Gemfile is beingprocessed, it defaults to "3.2.2".

How to install dependencies?

With the dual-boot setup in place, you can effectively manage dependencies forboth your current and next versions of the application.

Installing current dependencies: To install dependencies for your currentversion, simply run:

bundle install

This command uses the standard Gemfile to resolve and install dependencies foryour current application.

Installing next dependencies: To install dependencies for the next version ofyour application, use the following command:

BUNDLE_GEMFILE=Gemfile.next bundle install

This command specifies Gemfile.next as the Gemfile, allowing you to installdependencies specifically for the next version.

Managing commands for the next version: To perform various operations for thenext version, you can use the syntax BUNDLE_GEMFILE=Gemfile.next <command>.For example, to start the Rails server with the next version's dependencies, youwould use:

BUNDLE_GEMFILE=Gemfile.next rails s

This approach ensures that Gemfile.next is explicitly used for the specifiedcommand, allowing you to work with the next version of your application whilekeeping dependencies and configurations correctly managed.

Considerations for dual-boot setup

When implementing a dual-boot setup, it's crucial to consider where both Rubyand gem versions are defined. While gem versions are usually specified only inthe Gemfile, the Ruby version can be defined in several locations. The mostcommon are:

• .ruby-version File: Used by version managers like rbenv to specify the Rubyversion.
• Gemfile: This is where Bundler, RVM, Heroku, and similar tools reference theRuby version to manage the Ruby environment.
• Gemfile.lock: Although the Ruby version in this file is mostly informative,it's worth noting.
• Dockerfile: Defines the Ruby version for Docker containers if you're usingDocker.
• CI setup steps and configurations.
• Other less common locations within applications.

We need to make sure that dual-boot setup is compatible with all of them toensure a smooth transition and consistency across different environments.

Pre-compiled Ruby binaries

Pre-compiled Ruby binaries are distribution-ready versions of Ruby that includeoptimized features for specific systems. These Ruby binaries save time byeliminating the need to compile Ruby source code manually. Pre-compiled Rubybinaries help users quickly deploy applications that use different versions ofRuby on multiple machines.

RVM (Ruby Version Manager) is widely used for managing Rubyinstallations on Unix-like systems. RVM provides customized pre-compiled Rubybinaries tailored for various CPU architectures. These binaries offer additionalfeatures like readline support and SSL/TLS support. You can find them atRVM binaries.

The Need for pre-compiled Ruby Binaries

NeetoCI must execute user code in acontainerized environment. A Ruby environment is essential for running Ruby onRails applications. However, relying on the system's Ruby version is impracticalsince it may differ from the user's required version. Although rbenv or rvm canbe used to install the necessary Ruby version, this approach could be slow. Tosave time, we chose to leverage pre-compiled Ruby binaries.

As a CI/CD system, NeetoCI must ensure that all versions of Ruby that anapplication requires are always available. Hence, we decided to build ourbinaries instead of relying on binaries provided by RVM. Also, this would allowus to do more system-specific optimizations to the Ruby binary at build time.

Building pre-compiled Ruby binaries

We built a Ruby binary following theofficial documentation. We were able to execute it on our local development machines. But the samebinary ran into an error in our CI/CD environment.

$ bundle config path vendor/bundle./ruby: bad interpreter: No such file or directory

To debug the issue, we initially focused on $PATH. However, even afterresolving the $PATH issues, the problem persisted. We conducted a thoroughinvestigation to identify the root cause. Unfortunately, not much was written onthe Internet about this error. There was no mention of it in the officialRuby documentation.

As the next step, we decided to download the binary for version 3.2.2 fromRVM. While examining the configuration file, wenoticed that the following arguments were used with the configure command duringthe Ruby binary build process:

configure_args="'--prefix=/usr/share/rvm/rubies/ruby-3.2.2' '--enable-load-relative' '--sysconfdir=/etc' '--disable-install-doc' '--enable-shared'"

Here are the explanations of the configuration arguments:

1. --prefix=/usr/share/rvm/rubies/ruby-3.2.2: This specifies the directorywhere the Ruby binaries, libraries and other files will be kept after theinstallation is done.

2. --enable-load-relative: This specifies that Ruby can load relative pathsfor dynamically linked libraries. It allows the usage of relative pathsinstead of absolute paths when loading shared libraries. This feature can bebeneficial in specific deployment scenarios.

3. --sysconfdir=/etc: This argument sets the directory where Ruby's systemconfiguration files will be installed. In this case, it specifies the /etcdirectory as the location for these files.

4. --disable-install-doc: When this option is enabled, the installation ofdocumentation files during the build process is disabled. This can help speedup the build process and save disk space, especially if you do not requirethe documentation files.

5. --enable-shared: Enabling this option allows the building of sharedlibraries for Ruby. Shared libraries enable Ruby to dynamically link and loadspecific functionality at runtime, leading to potential performanceimprovements and reduced memory usage.

In simpler terms, when the --enable-load-relative flag is enabled, thecompiled Ruby binary can search for shared libraries in its own directory usingthe $ORIGIN variable.

When I built the binary on the Docker registry, then the passed --prefix wassomething like /usr/share/neetoci. When the binary is built, then binary had/usr/share/neetoci is hard-coded at various places. When we download thisbinary and use in CI then in the CI environment, Ruby is looking for/user/share/neetoci to load dependencies.

By enabling --enable-load-relative flag while building the binary Ruby willnot use the hard coded value. Rather Ruby will use $ORIGIN variable and willsearch for the dependencies in the directory mentioned in $ORIGIN.

This is particularly helpful when the Ruby binary is relocated to a differentdirectory or system. By using relative paths with $ORIGIN, the binary can findits shared libraries regardless of its new location. Without this flag, sharedlibraries are loaded using absolute paths, which can cause issues if the binaryis moved to a different location and cannot locate its shared libraries.

In our specific use case, where we create and download binaries in separatecontainers, we encountered an error due to the absolute paths. To overcome this,we enabled the --enable-load-relative flag. This allowed the binary to findits shared libraries successfully, and it worked as expected in our CI/CDenvironment.

To display test suites in a tree structure, we need to store some informationabout the parent-child relationship in the database. This is whereAncestry comes in. Ancestry is a gemthat allows Rails Active Record models to be organized as a tree structure.

Normally, web applications implement pagination to show a list of records. But,implementing pagination for a tree-structured data can be challenging and canmake the application more complex. To avoid pagination, the application musthave enough performance to display an entire tree having a large number ofnodes/records without significant delays.

In this blog, we will discuss on how we leveraged the full potential of theAncestry gem to address the performance issues encountered while listing suites.

1. Migration to materialized_path2

There are several ways to store hierarchical data in a relational database, suchas materialized paths, closure tree tables, adjacency lists, and nested sets.Ancestry Gem uses the materialized path pattern to store hierarchical data.

The materialized path pattern is a technique in which a single node is stored inthe database as a record and it has an additional column to store thehierarchical information. In the case of the Ancestry gem, this additionalcolumn is named as ancestry. The ancestry column is used to store IDs of theancestors of a node as a single string, separated by a delimiter.

In order to understand how Ancestry gem uses materialized path pattern, firstlet's look at the nodes we have in our example. In the screenshot posted above,we see the following four nodes:

In our example, the s1 is a root node, s11 and s12 are the children of node s1,and s111 is the child of s11.

In order to store the hierarchical data, the gem offers two types of ancestryformats, materialized_path and materialized_path2. In these techniques, eachnode is represented by a record in the database. Our example consists of fournodes, so there will be four records in the database. The only differencebetween materialized_path and materialized_path2 lies in the format inwhich, the IDs are stored in the ancestry column.

materialized_path

Here the IDs of ancestors are stored in the format "id-1/id-2/id-3", wereid-1, id-2 and id-3 are the IDs of three nodes with / as the delimiter.The id-1 is the root node, id-2 is the child of id-1 and id-3 is thechild of id-2. In case of a root node, the ancestry will be null.

The table below shows how the suites in our example are stored in the databaseusing materialized_path:

This arrangement of node IDs as a single string makes it easier to query for alldescendants of a node, as we use SQL string functions to match the ancestrycolumn. Here is the SQL statement to get descendants of suite s1:

SELECT "suites".* FROM "suites" WHERE ("suites"."ancestry" LIKE 's1/%' OR "suites"."ancestry" = 's1')

The result of the above query is:

materialized_path2

materialized_path2 stores ancestors in the format "/id-1/id-2/id-3/", wereid-1 is the root node, id-2 is the child of id-1, and id-3 is the childof id-2. Here the delimiter is / as same as materialized_path, but theancestry will be starting with a / and ending with a /. For a root nodethe ancestry will be /.

The table below shows how the suites in our example are stored in the databaseusing materialized_path2:

The SQL statement to get the descendants of suite s1 is:

SELECT "suites".* FROM "suites" WHERE "suites"."ancestry" LIKE '/s1/%'

The result of above query is:

If we compare the 2 SQL queries, we can see that materialized_path2 has oneless "OR" statement. This gives materialized_path2 a slight advantage inperformance.

In NeetoTestify, we previously used the materialized_path format, but now wehave migrated to materialized_path2 for improved performance.

2. Added collation to the ancestry column

CollationIn database systems, the index specifies how data is sorted and compared in adatabase. Collation provides the sorting rules, case sensitivity, and accentsensitivity properties for the data in the database.

As mentioned above, our resulting query for fetching the descendants of a nodewould be

SELECT "suites".* FROM "suites" WHERE "suites"."ancestry" LIKE '/s1/%'

It uses a LIKE query for comparison with a wildcard character(%). In general,when using the LIKE query with a wildcard character (%) on the right-hand sideof the pattern, the database can utilize an index and potentially optimize thequery performance. This optimization holds true for ASCII strings.

However, it's important to note that this optimization may not necessarily holdtrue for Unicode strings, as Unicode characters can have varying lengths anddifferent sorting rules compared to ASCII characters.

In our case, the ancestry column contains only ASCII strings. If we let thedatabase know about this constraint, we can optimize the database's queryperformance. To do that, we need to specify the collation type of the ancestrycolumn.

FromPostgres's documentation:

The C and POSIX collations both specify traditional C behavior, in whichonly the ASCII letters A through Z are treated as letters, and sorting isdone strictly by character code byte values.

Since we are using Postgres in NeetoTestify, we use collation C. Instead if weuse MySQL, then the ancestry suggests using collation utf8mb4_bin.

class AddAncestryToTable < ActiveRecord::Migration[6.1]  def change    change_table(:table) do |t|      # postgres      t.string "ancestry", collation: 'C', null: false      t.index "ancestry"      # mysql      t.string "ancestry", collation: 'utf8mb4_bin', null: false      t.index "ancestry"    end  endend

3. Usage of arrange method

Previously, we were constructing the tree structure of the suites by fetchingthe children of each node individually from the database. For this, we firstfetched the suites whose ancestry is / (root nodes). Then, for each of thesesuites, we fetched their children, and repeated this process until we reachedleaf-level suites.

This recursive approach results in a large number of database queries, causingperformance issues as the tree size increases. Constructing a tree with n(4)nodes required n+1(5) database queries, adding to the complexity of the process.

The arrange method provided by Ancestry gem, converts the array of nodes intonested hashes, utilizing the ancestry column information. Also by using thismethod the number of database queries will remain 1, even if the number ofsuites and nested suites increases.

suites = project.suites# SELECT "suites".* FROM "suites" WHERE "suites"."project_id" = "p1"# [#   <Suite id: "s1", project_id: "p1", name: "Suite 1", ancestry: "/">,#   <Suite id: "s11", project_id: "p1", name: "Suite 1.1", ancestry: "/s1/">,#   <Suite id: "s111", project_id: "p1", name: "Suite 1.1.1", ancestry: "/s1/s11/">,#   <Suite id: "s12", project_id: "p1", name: "Suite 1.2", ancestry: "/s1/"># ]suites.arrange# {#   <Suite id: s1, project_id: "p1", name: "Suite 1", ancestry: "/"> => {#     <Suite id: s11, project_id: "p1", name: "Suite 1.1", ancestry: "/s1/"> => {#       <Suite id: s111, project_id: "p1", name: "Suite 1.1.1", ancestry: "/s1/s11/"> => {}#     },#     <Suite id: s12, project_id: "p1", name: "Suite 1.2", ancestry: "/s1/"> => {}#   }# }

The recursive approach took 5.72 seconds to retrieve 170 suites, but the arrayconversion approach of arrange method, retrieved the same number of suites in728.72 ms.

The above image shows the advantage in performance while using the arrangemethod over the recursive approach. For the comparison of two approaches 170suites and 1650 test cases were considered.

By implementing the above 3 best practices, we were able to considerably improvethe overall performance of the application.

All three videos were prepared for my team members to show how to go aboutdebugging. The videos are being presented "as it was recorded".

First Case Study

When a job fails in Sidekiq, Sidekiq puts that job inRetrySet and retries thatjob until the job succeeds or the job reaches the maximum number of retries. Bydefault the maximum number of retries is 25. If a job fails 25 times, then thatjob is moved to the DeadSet.By default, Sidekiq will store up to 10,000 jobs in the deadset.

We had a situation where Redis was running out of memory. Here is how thedebugging was done.

<iframewidth="560"height="315"src="https://www.youtube.com/embed/dg-K_IoT-x0"title="YouTube video player"frameborder="0"allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"allowfullscreen

</iframe>

How to inspect the deadset

ds = Sidekiq::DeadSet.newds.each do |job|  puts "Job #{job['jid']}: #{job['class']} failed at #{job['failed_at']}"end

Running the following to view the latest entry to the dataset:

ds.firstds.count

To see the memory usage following commands were executed in the Redis console.

> memory usage dead30042467> type deadzset

As discussed in the video large amount of payload was being sent. This is notthe right way to send data to the worker. Ideally, some sort of id should besent to the worker and the worker should be able to get the necessary data fromthe database based on the received id.

References

1. How to increase the number of jobs in the Sidekiq deadset or disable deadset
2. Maximum number of job retries in Sidekiq
3. Maximum number of jobs in Sidekiq Deadset

Second case study

In this case, the Redis instance of neetochatwas running out of memory. The Redis instance had 50MB capacity, but we weregetting the following error.

ERROR: heartbeat: OOM command not allowed when used memory > 'maxmemory'.

We were pushing too many geo info records to Redis and that caused the memory tofill up. Here is the video capturing the debugging session.

<iframewidth="560"height="315"src="https://www.youtube.com/embed/oz7Pcbc_zxM"title="YouTube video player"frameborder="0"allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"allowfullscreen

</iframe>

The following are the commands executed while debugging.

> pingPONG> info> info memory> info keyspace> keys *failed*> keys *process*> keys *geocoder*> get getocoder:http://ipinfo.io/41.174.30.55/geo?

Third Case Study

In this case, the authentication service of Neeto wasfailing because of memory exhaustion.

Here the number of keys was limited, but the payload data was huge and all thatpayload data was hogging the memory. Here is the video capturing the debuggingsession.

<iframewidth="560"height="315"src="https://www.youtube.com/embed/a_Ygbcreokw"title="YouTube video player"frameborder="0"allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"allowfullscreen

</iframe>

The following are the commands executed while debugging.

> ping> info keyspacedb0:keys=106, expires=86,avg_ttl=1233332728573> key * (to see all the keys)

The last command listed all 106 keys. Next, we needed to find how much memoryeach of these keys are using. For that, the following commands were executed.

> memory usage organizations/subdomains/bigbinary/neeto_app_links736 bytes> memory usage failed10316224 (10MB)> memory usage dead29871174 (29MB)

We can see many implementations for calculating all subclasses of a particularclass from the Ruby community with different gems. TheActiveSupport::DescendantsTrackeris one of such implementations used in Rails framework. Finally, Ruby has addedthe Class#subclasses native implementation for it's 3.1 version release.

After Ruby 3.1

=> class User; end=> class Employee < User; end=> class Client < User; end=> class Manager < Employee; end=> class Developer < Employee; end=> User.subclasses=> [Employee, Client]=> Employee.subclasses=> [Manager, Developer]=> Developer.subclasses=> []

Here's the relevant pull request andfeature discussion for this change.

=> str = "Ruby 3.1 introduces MatchData#match method"=>  m = /(?<lang>\S+)(\s)(?<version>[\d\.]+)/.match(str)=> #<MatchData "Ruby 3.1" lang:"Ruby" version:"3.1">

Before Ruby 3.1

We have MatchData#[] method to access all the substring groups that arematched. We can access either a single match or multiple matches by giving asingle index or a range of indexes, respectively, using MatchData#[] method.

=> m[:lang]=> "Ruby"=> m[1]=> "Ruby"=> m[:version]=> "3.1"=> m[0..2]=> ["Ruby 3.1", "Ruby", "3.1"]=> m[1..2]=> ["Ruby", "3.1"]=> m[0].length=> 8

After Ruby 3.1

We have two new methods now to access the match data & its length. TheMatchData#match & MatchData#match_length methods accepts either an index ora symbol as an argument to return the match data & its length.

=> m.match(0)=> "Ruby 3.1"=> m.match(:version)=> "3.1"=> m.match(3)=> nil=> m.match_length(0)=> 8=> m.match_length(:version)=> 3=> m.match_length(3)=> nil

Please note that MatchData#match is not same as MatchData#[] method. Thelater accepts an optional length or range as an argument to return the matcheddata but the former doesn't accept it, instead it allows only a single index orsymbol as an argument.

Here's the relevant pull request andfeature discussion for this change.

We already knowArray#intersection or Array#&methods which are used to find the common elements between arrays.

=> x = [1, 2, 5, 8]=> y = [2, 4, 5, 9]=> z = [3, 7]=> x.intersection(y) # x & y=> [2, 5]=> x.intersection(z) # x & z=> []

The intersection or & methods return an empty array or array having thecommon elements in it as result. We have to further call empty?, any? orblank? like methods to check whether two arrays intersect each other or not.

Before Ruby 3.1

=> x.intersection(y).empty?=> false=> (x & z).empty?=> true=> (y & z).any?=> false

After Ruby 3.1

=> x.intersect?(y)=> true=> y.intersect?(z)=> false

The Array#intersect? method accepts only single array as argument, butArray#intersection method can accept multiple arrays as arguments.

=> x.intersection(y, z) # x & y & z=> []

The newly introduced intersect? method is faster than the above describedchecks using intersection or & since the new method avoids creating anintermediate array while evaluating for common elements. Also new method returnstrue as soon as it finds a common element between arrays.

Here's the relevant pull request andfeature discussion for this change.

Ruby 3.1 introduces an optional hash argument for the Enumerable#tally methodto count. If a hash is given, the total number of occurrences of each element isadded to the hash values and the final hash is returned.

Ruby 2.7.0+

=> letters = ["a", "b", "c", "a", "d", "c", "a", "c", "a"]=> result = letters.tally=> {"a"=>4, "b"=>1, "c"=>3, "d"=>1}

Before Ruby 3.1

=> new_letters = ["a", "b", "c", "a", "c", "a"]=> new_letters.tally(result)=> ArgumentError (wrong number of arguments (given 1, expected 0))

After Ruby 3.1

=> new_letters = ["a", "b", "c", "a", "c", "a"]=> new_letters.tally(result)=> {"a"=>7, "b"=>2, "c"=>5, "d"=>1}

The value corresponding to each element in the hash must be an integer.Otherwise, the method raises TypeError on execution.

If the default value is defined for the given hash, it will be ignored and thecount of occurrences will be added in the returned hash.

=> letters = ["a", "b", "c", "a"]=> letters.tally(Hash.new(10))=> {"a"=>2, "b"=>1, "c"=>1}

Here's the relevant pull request andfeature discussion for this change.

Ruby 3.1 introduces the compact method in the Enumerable module. Now we canuse the compact method along with the Enumerator and Enumerator::Lazyclasses which include the Enumerable module.

Before Ruby 3.1

=> enum = [1, nil, 3, nil, 5].to_enum=> #<Enumerator: ...>=> enum.compact=> NoMethodError (undefined method `compact' for #<Enumerator: [1, nil, 3, nil, 5]:each>)=>  enum.reject { |x| x.nil? }=> [1, 3, 5]

After Ruby 3.1

=> enum = [1, nil, 3, nil, 5].to_enum=> #<Enumerator: ...>=> enum.compact=> [1, 3, 5]

We can access the compact method to remove all nil occurrences from anyclasses where we include the Enumerable module.

class Person  include Enumerable  attr_accessor :names  def initialize(names = [])    @names = names  end  def each &block    @names.each(&block)  endend=> list = Person.new(["John", nil, "James", nil])=> #<Person:0x0000000101cd3de8 @names=["John", nil, "James", nil]>=> list.compact=> ["John", "James"]

Similarly, lazy evaluation can be chained with the compact method to removeall nil entries from the Enumerator collection.

=> enum = [1, nil, 3, nil, 5].to_enum.lazy.compact=> #<Enumerator::Lazy: ...>=> enum.force=> [1, 3, 5]=> list = Person.new(["John", nil, "James", nil]).lazy.compact=> #<Enumerator::Lazy: ...>=> list.force=> ["John", "James"]

Here's the relevant pull request andfeature discussion for this change.

Ruby 3 major updates

The number 3 is very significant in the Ruby 3 release. Be it releaseversion number, making performance 3x faster, or the trio of corecontributors(Matz, TenderLove, Koichi). Similarly, there were 3 major goals ofRuby 3: being faster, having better concurrency, and ensuring correctness.

1. Ruby 3 performance

One of the major focuses for Ruby 3 was the performance. In fact, the initialdiscussion for Ruby 3 was started around it. Matz had set a very ambitious goalof making Ruby 3 times faster.

What is Ruby 3x3?

Before discussing this, let's revisit Ruby's core philosophy.

"I hope to see Ruby help every programmer in the world to be productive, andto enjoy programming, and to be happy." - Matz

About Ruby 3x3, some asked whether the goal was to make Ruby the fastestlanguage? The answer is no. The main goal of Ruby 3x3 was to make Ruby 3 timesfaster than Ruby 2.

No language is fast enough. - Matz

Ruby was not designed to be fastest and if it would have been the goal, Rubywouldn't be the same as it is today. As Ruby language gets performance boost, itdefinitely helps our application be faster and scalable.

"In the design of the Ruby language we have been primarily focused onproductivity and the joy of programming. As a result, Ruby was too slow." -Matz

There are two areas where performance can be measured: memory and CPU.

CPU optimization

Some enhancements in Ruby internals have been made to improve the speed. TheRuby team has optimized the JIT(Just In Time) compiler from previous versions.Ruby MJIT compiler wasfirst introduced in Ruby 2.6. Ruby 3 MJIT comes with better security and seemsto improve web apps performance to a greater extent.

MJIT implementation is different from the usual JIT. When methods get calledrepeatedly e.g. 10000 times, MJIT will pick such methods which can be compiledinto native code and put them into a queue. Later MJIT will fetch the queue andconvert them to native code.

Please checkJIT vs MJITfor more details.

Memory optimization

Ruby 3 comes with an enhanced garbage collector. It has python's buffer-like APIwhich helps in better memory utilization. Since Ruby 1.8, Ruby has continuouslyevolved inGarbage collectionalgorithms.

Automatic Garbage Compaction

The latest change in garbage collection isGarbage Compaction.It was introduced in Ruby 2.7 where the process was a bit manual. But in version3 it is fully automatic. The compactor is invoked aptly to make sure propermemory utilization.

Objects Grouping

The garbage compactor moves objects in the heap. It groups dispersed objectstogether at a single place in the memory so that this memory can be used even byheavier objects later.

2. Parallelism and Concurrency in Ruby 3

Concurrency is one of the important aspects of any programming language. Matzfeels that Threads are not the right level of abstraction for Ruby programmersto use.

I regret adding Threads. - Matz

Ruby 3 makes it a lot easier to make applications where concurrency is a majorfocus. There are several features and improvements added in Ruby 3 related toconcurrency.

Fibers

Fibers are considered a disruptive addition in Ruby 3. Fibers are light-weightworkers which appear like Threads but have some advantages. It consumes lessmemory than Threads. It gives greater control to the programmer to define codesegments that can be paused or resumed resulting in better I/O handling.

Falcon Rack web server uses Async Fibersinternally. This allows Falcon to not block on I/O. Asynchronously managing I/Ogives a great uplift to the Falcon server to serve requests concurrently.

Fiber Scheduler

Fiber Scheduler is an experimentalfeature added in Ruby 3. It was introduced to intercept blocking operations suchas I/O. The best thing is that it allows lightweight concurrency and can easilyintegrate into the existing codebase without changing the original logic. It'san interface and that can be implemented by creating a wrapper for a gem likeEventMachine or Async. This interface design allows separation of concernsbetween the event loop implementation and the application code.

Following is an example to send multiple HTTP requests concurrently usingAsync.

require 'async'require 'net/http'require 'uri'LINKS = [  'https://bigbinary.com',  'https://basecamp.com']Async do  LINKS.each do |link|    Async do      Net::HTTP.get(URI(link))    end  endend

Please check fibers formore details.

Ractors (Guilds)

As we know Rubys global VM lock (GVL) prevents most Ruby Threads fromcomputing in parallel.Ractors work aroundthe GVL to offer better parallelism. Ractor is an Actor-Model like concurrentabstraction designed to provide a parallel execution without thread-safetyconcerns.

Ractors allows Threads in different Ractors to compute at the same time. EachRactor has at least one thread, which may contain multiple fibers. In a Ractor,only a single thread is allowed to execute at a given time.

The following program returns the square root of a really large number. Itcalculates the result for both numbers in parallel.

# Math.sqrt(number) in ractor1, ractor2 run in parallelractor1, ractor2 = *(1..2).map do  Ractor.new do    number = Ractor.recv    Math.sqrt(number)  endend# send parametersractor1.send 3**71ractor2.send 4**51p ractor1.take #=> 8.665717809264115e+16p ractor2.take #=> 2.251799813685248e+15

3. Static Analysis

We need tests to ensure correctness of our program. However by its very naturetests could mean code duplication.

I hate tests because they aren't DRY. - Matz

To ensure the correctness of a program, static analysis can be a great tool inaddition to tests.

The static analysis relies on inline type annotations which aren't DRY. Thesolution to address this challenge is having .rbs files parallel to our .rbfiles.

RBS

RBS is a language to describe the structure of a Ruby program. It provides us anoverview of the program and how overall classes, methods, etc. are defined.Using RBS, we can write the definition of Ruby classes, modules, methods,instance variables, variable types, and inheritance. It supports commonly usedpatterns in Ruby code, and advanced types like unions and duck typing.

The .rbs files are something similar to .d.ts files in TypeScript. Followingis a small example of how a .rbs file looks like. The advantage of having atype definition is that it can be validated against both implementation andexecution.

The below example is pretty self-explanatory. One thing we can note here though,each_post accepts a block or returns an enumerator.

# user.rbsclass User  attr_reader name: String  attr_reader email: String  attr_reader age: Integer  attr_reader posts: Array[Post]  def initialize: (name: String,                   email: String,                   age: Integer) -> void  def each_post: () { (Post) -> void } -> void                   | () -> Enumerator[Post, void]end

Please check RBS gem documentation for moredetails.

Typeprof

Introducing type definition was a challenge because there is already a lot ofexisting Ruby code around and we need a tool that could automatically generatethe type signature. Typeprof is a type analysis tool that reads plain Ruby codeand generates a prototype of type signature in RBS format by analyzing themethods, and its usage. Typeprof is an experimental feature. Right now onlysmall subset of ruby is supported.

Ruby is simple in appearance, but is very complex inside, just like our humanbody. - Matz

Let's see an example.

# user.rbclass User  def initialize(name:, email:, age:)    @name, @email, @age = name, email, age  end  attr_reader :name, :email, :ageendUser.new(name: "John Doe", email: 'john@example.com', age: 23)

Output

$ typeprof user.rb# Classesclass User  attr_reader name : String  attr_reader email : String  attr_reader age : Integer  def initialize : (name: String,                    email: String,                    age: Integer) -> [String, String, Integer]end

Other Ruby 3 features and changes

In the 7-year period, the Ruby community has seen significant improvement inperformance and other aspects. Apart from major goals, Ruby 3 is an excitingupdate with lots of new features, handy syntactic changes, and new enhancements.In this section, we will discuss some notable features.

We are making Ruby even better. - Matz

One-line pattern matching syntax change

Previously one-line pattern matching used the keyword in. Now it's replacedwith =>.

Ruby 2.7

  { name: 'John', role: 'CTO' } in {name:}  p name # => 'John'

Ruby 3.0

  { name: 'John', role: 'CTO' } => {name:}  p name # => 'John'

Find pattern

Thefind patternwas introduced in Ruby 2.7 as an experimental feature. This is now part ofRuby 3.0. It is similar to pattern matching in Elixir or Haskell.

users = [  { name: 'Oliver', role: 'CTO' },  { name: 'Sam', role: 'Manager' },  { role: 'customer' },  { name: 'Eve', city: 'New York' },  { name: 'Peter' },  { city: 'Chicago' }]users.each do |person|  case person  in { name:, role: 'CTO' }    p "#{name} is the Founder."  in { name:, role: designation }    p "#{name} is a #{designation}."  in { name:, city: 'New York' }    p "#{name} lives in New York."  in {role: designation}    p "Unknown is a #{designation}."  in { name: }    p "#{name}'s designation is unknown."  else    p "Pattern not found."  endend"Oliver is the Founder.""Sam is a Manager.""Unknown is a customer.""Eve lives in New York.""Peter's designation is unknown.""Pattern not found."

Endless Method definition

This is another syntax enhancement that is optional to use. It enables us tocreate method definitionswithout end keyword.

def: increment(x) = x + 1p increment(42) #=> 43

Except method in Hash

Sometimes while working on a non Rails app I get undefined method except. Theexcept method was available only in Rails. In Ruby 3 Hash#except wasadded to Rubyitself.

user = { name: 'Oliver', age: 29, role: 'CTO' }user.except(:role) #=> {:name=> "Oliver", :age=> 29}

Memory View

This is again an experimental feature. This is a C-API that will allow extensionlibraries to exchange raw memory area. Extension libraries can also sharemetadata of memory area that consists of shape and element format. It wasinspired byPythons buffer protocol.

Arguments forwarding

Arguments forwarding (...) now supports leading arguments.

It is helpful in method_missing, where we need method name as well.

def method_missing(name, ...)  if name.to_s.end_with?('?')    self[name]  else    fallback(name, ...)  endend

Other Notable changes

• Pasting in IRB is much faster.
• The order of backtrace had beenreversed.The error message and line number are printed first, rest of the backtrace isprinted later.
• Hash#transform_keysaccepts a hash that maps old keys with new keys.
• Interpolated String literals are no longer frozen when# frozen-string-literal: true is used.
• Symbol#to_proc now returns a lambda Proc.
• Symbol#name has beenadded, which returns the symbol's name as a frozen string.

Many other changes can be checked atRuby 3 News formore details.

Transition

A lot of core libraries have been modified to fit the Ruby 3 goal needs. Butthis doesn't mean that our old applications will suddenly stop working. The Rubyteam has made sure that these changes are backward compatible. We might see somedeprecation warnings in our existing code. The developers can fix these warningsto smoothly transition from an old version to the new version. We are all set touse new features and get all new performance improvements.

Conclusion

With great improvements in performance, memory utilization, static analysis, andnew features like Ractors and schedulers, we have great confidence in the futureof Ruby. With Ruby 3, the applications can be more scalable and more enjoyableto work with. The coming 2021 is not just a new year but rather a new era forall Rubyists. We at BigBinary thank everyone who contributed towards the Ruby 3release directly or indirectly.

Happy Holidays and Happy New Year folks!!!

Usage before Ruby 3

The following example shows how we used to apply transform_keys:

# 1. Declare address hashirb(main)> address = {House: 'Kizhakkethara', house_no: 123, locality: 'India'}=> {:House=>"Kizhakkethara", :house_no=>123, :locality=>"India"}# 2. Lowercase all the keysirb(main)> address.transform_keys(&:downcase)=> {:house=>"Kizhakkethara", :house_no=>123, :locality=>"India"}# 3. Replace a particular key with a new key along with lowercasingirb(main)* address.transform_keys do |key|irb(main)*   new_key = keyirb(main)*   if key == :localityirb(main)*     new_key = :countryirb(main)*   endirb(main)*   new_key.to_s.downcase.to_symirb(main)> end=> {:house=>"Kizhakkethara", :house_no=>123, :country=>"India"}

Although the changes required are trivial,we ended up writing a block to do the job.But what happens when the number of keysthat needs to be transformed increases?Do we need to write n-number of conditions within a block?Not anymore!

Introducing Hash#transform_keys with hash argument

Let's take the same example and provide a hash,which will be used for the transformation:

# 1. Declare address hashirb(main)> address = {House: 'Kizhakkethara', house_no: 123, locality: 'India'}=> {:House=>"Kizhakkethara", :house_no=>123, :locality=>"India"}# 2. Provide hash with transform_keysirb(main)> address.transform_keys({House: :house, locality: :country})=> {:house=>"Kizhakkethara", :house_no=>123, :country=>"India"}

That does the job.But let's try to improve this code.Ultimately what happens when we invoke that methodis that it goes througheach of the keys in our variableandmaps the existing keys to the new keys.The transform_keys method accepts a block as a parameter.Thus let's pass in the downcase method as a Proc argument:

# 1. Passing in block parametersirb(main)> address.transform_keys({locality: :country}, &:downcase)=> {:house=>"Kizhakkethara", :house_no=>123, :country=>"India"}

An important point to be noted about the block parameter is that,it's only applied to keys which are not specified in the hash argument.

Other common use cases

Transforming params received in the Rails controller

# 1. Declare paramsirb(rails)> params = ActionController::Parameters.new({"firstName"=>"oliver", "lastName"=>"smith", "email"=>"oliver@bigbinary.com"})=> <ActionController::Parameters {"firstName"=>"oliver", "lastName"=>"smith", "email"=>"oliver@bigbinary.com"} permitted: false># 2. Convert camelCase to snake_case using block parameterirb(rails)> params.permit(:firstName, :lastName, :email).transform_keys(&:underscore)=> <ActionController::Parameters {"first_name"=>"oliver", "last_name"=>"smith", "email"=>"oliver@bigbinary.com"} permitted: true># 3. Or using hash argumentirb(rails)> params.permit(:firstName, :lastName, :email).transform_keys({firstName: 'first_name', lastName: 'last_name'})=> <ActionController::Parameters {"first_name"=>"oliver", "last_name"=>"smith", "email"=>"oliver@bigbinary.com"} permitted: true>

Slicing hash along with key transformation

irb(main)> address.transform_keys({locality: :country}).slice(:house_no, :country)=> {:house_no=>123, :country=>"India"}

Transforming keys in place using bang counterpart

irb(main)> address.transform_keys!({locality: :country}, &:downcase)irb(main)> address=> {:house=>"Kizhakkethara", :house_no=>123, :country=>"India"}

References

• Discussions regarding this feature can be foundhere.
• Commit for this feature can be foundhere.

Why do we need Hash#except?

At times,we needto print or log everythingexcept some sensitive data.Let's saywe want to printuser detailsin the logsbut not passwords.

Before Ruby 3 we could have achievedit in the following ways:

irb(main):001:0> user_details = { name: 'Akhil', age: 25, address: 'India', password: 'T:%g6R' }# 1. Reject with a block and include?irb(main):003:0> puts user_details.reject { |key, _| key == :password }=> { name: 'Akhil', age: 25, address: 'India' }# 2. Clone the hash with dup, tap into it and delete that key/value from the cloneirb(main):005:0> puts user_details.dup.tap { |hash| hash.delete(:password) }=> { name: 'Akhil', age: 25, address: 'India' }

We know that ActiveSupport alreadycomes with Hash#exceptbut for a simple Ruby applicationusing ActiveSupport wouldbe overkill.

Ruby 3

To make the above taskeasier and more explicit,Ruby 3 adds Hash#exceptto return a hashexcluding the given keys and their values:

irb(main):001:0> user_details = { name: 'Akhil', age: 25, address: 'India', password: 'T:%g6R' }irb(main):002:0> puts user_details.except(:password)=> { name: 'Akhil', age: 25, address: 'India' }irb(main):003:0> db_info = YAML.safe_load(File.read('./database.yml'))irb(main):004:0> puts db_info.except(:username, :password)=> { port: 5432, database_name: 'example_db_production' }

Check out thecommitfor more details.Discussion around it can be found here.

One of the important aspects of Ruby 3.0 is optimization.The part of that optimization isthe introduction of name method for Symbol.In this blog,we will take a lookat what name method of class Symbol doesandwhy it was introduced.The new name method is introduced on Symbolto simply convert a symbol into a string.Symbol#name returns a string.Let's see how it works.

irb(main):001:0> :simba.name=> 'simba'irb(main):002:0> :simba.name.class=> Stringirb(main):003:0> :simba.name === :simba.name=> true

Wait what?Don't we have to_sto convert a symbol into a string.Most of us have used to_s method on a Symbol.The to_s method returns a String objectandwe can simply use it.But why name?

Using to_s is okay in most cases.But the problem with to_s isthat it creates a new String objectevery time we call it on a symbol.We can verify this in irb.

irb(main):023:0> :simba.to_s.object_id=> 260irb(main):024:0> :simba.to_s.object_id=> 280

Creating a new object for every symbol to a string conversionallocates new memory which increases overhead.The light was thrown on this issue byschneems (Richard Schneeman)in a talk at RubyConf Thailandwhere he showed how Symbol#to_s allocationcauses significant overhead in ActiveRecord.This inspired Ruby communityto have a new method name on Symbolwhich returns a frozen string object.This reduces the string allocations dramaticallywhich results in reducing overhead.

irb(main):001:0> :simba.name.frozen?=> trueirb(main):002:0> :simba.name.object_id=> 200irb(main):003:0> :simba.name.object_id=> 200

The reasonto bring this feature wasthat most of the timeswe want a simple string representationfor displaying purposeorto interpolate into another string.The result of to_s is rarely mutated directly.By introducing this methodwe save a lot of objectswhich helps in optimization.Now we know the benefits of name,we should prefer using name over to_swhen we don't want to mutate a string.

For more information on discussion,official documentation,please head on to Feature #16150 discussion,Pull requestandRuby 3.0 official release preview.

# endless method definition>> def raise_to_power(number, power) = number ** power>> raise_to_power(2, 5)=> 32

The discussion aroundit can be found here.Check out thepull requestfor more details on this.

(..100) is a Beginless Range and it is equivalent to (nil..100).

Let's see how Beginless Range could be used.

> array = (1..10).to_a# Select first 6 elements> array[..5]=> [1, 2, 3, 4, 5, 6]# Select first 5 elements> array[...5]=> [1, 2, 3, 4, 5]# grep (INFINITY..5) in (1..5)> (1..10).grep(..5)=> [1, 2, 3, 4, 5]# (..100) is equivalent to (nil..100)> (..100) == (nil..100)=> true

Here is another example where in the case statement the condition can be readas below the specified level.

case temperaturewhen ..-15  puts "Deep Freeze"when -15..8  puts "Refrigerator"when 8..15  puts "Cold"when 15..25  puts "Room Temperature"when (25..)   # Kindly notice the brackets here  puts "Hot"end

It can also be used for defining constants for ranges.

TEMPERATURE = {  ..-15  => :deep_freeze,  -15..8 => :refrigerator,  8..15  => :cold,  15..25 => :room_temperature,  25..   => :hotend

Using Beginless Range in DSL makes it easier to write conditions and it looksmore natural.

# In RailsUser.where(created_at: (..DateTime.now))# User Load (2.2ms)  SELECT "users".* FROM "users" WHERE "users"."created_at" <= $1 LIMIT $2  [["created_at", "2020-08-05 15:00:19.111217"], ["LIMIT", 11]]# In RubySpecruby_version(..'1.9') do# Tests for old Rubyend

Here is the relevantcommitand discussion regarding this change.

Before Ruby 2.7, we could have achieved the same with 2 iterations usingselect & map combination or map & compact combination.

irb> numbers = [3, 6, 7, 9, 5, 4]# we can use select & map to find square of odd numbersirb> numbers.select { |x| x.odd? }.map { |x| x**2 }=> [9, 49, 81, 25]# or we can use map & compact to find square of odd numbersirb> numbers.map { |x| x**2 if x.odd? }.compact=> [9, 49, 81, 25]

Ruby 2.7

Ruby 2.7 adds Enumerable#filter_map which can be used to filter & map theelements in a single iteration and which is more faster when compared to otheroptions described above.

irb> numbers = [3, 6, 7, 9, 5, 4]irb> numbers.filter_map { |x| x**2 if x.odd? }=> [9, 49, 81, 25]

The original discussion had started8 years back. Here is the latestthread andgithub commitfor reference.

Ruby 2.7 NEWShas listed the spec of keyword arguments for Ruby 3.0. We will take the examplesmentioned there and for each scenario we will look into how we can fix them inthe existing codebase.

Scenario 1

When method definition accepts keyword arguments as the last argument.

def sum(a: 0, b: 0)  a + bend

Passing exact keyword arguments in a method call is acceptable, but inapplications we usually pass a hash to a method call.

sum(a: 2, b: 4) # OKsum({ a: 2, b: 4 }) # Warned

In this case, we can add a double splat operator to the hash to avoiddeprecation warning.

sum(**{ a: 2, b: 4 }) # OK

Scenario 2

When method call passes keyword arguments but does not pass enough requiredpositional arguments.

If the number of positional arguments doesn't match with method definition, thenkeyword arguments passed in method call will be considered as the lastpositional argument to the method.

def sum(num, x: 0)  num.values.sum + xend

sum(a: 2, b: 4) # Warnedsum(a: 2, b: 4, x: 6) # Warned

To avoid deprecation warning and for code to be compatible with Ruby 3, weshould pass hash instead of keyword arguments in method call.

sum({ a: 2, b: 4 }) # OKsum({ a: 2, b: 4}, x: 6) # OK

Scenario 3

When a method accepts a hash and keyword arguments but method call passes onlyhash or keyword arguments.

If a method arguments are a mix of symbol keys and non-symbol keys, and themethod definition accepts either one of them then Ruby splits the keywordarguments but also raises a warning.

def sum(num={}, x: 0)  num.values.sum + xend

sum("x" => 2, x: 4) # Warnedsum(x: 2, "x" => 4) # Warned

To fix this warning, we should pass hash separately as defined in the methoddefinition.

sum({ "x" => 4 }, x: 2) # OK

Scenario 4

When an empty hash with double splat operator is passed to a method thatdoesn't accept keyword arguments.

Passing keyword arguments using double splat operator to a method that doesn'taccept keyword argument will send empty hash similar to earlier version of Rubybut will raise a warning.

def sum(num)  num.values.sumend

numbers = {}sum(**numbers) # Warned

To avoid this warning, we should change method call to pass hash instead ofusing double splat operator.

numbers = {}sum(numbers) # OK

Added support for non-symbol keys

In Ruby 2.6.0, support for non-symbol keys in method call was removed. It isadded back in Ruby 2.7. When method accepts arbitrary keyword arguments usingdouble splat operator then non-symbol keys can also be passed.

def sum(**num)  num.values.sumend

ruby 2.6.5

sum("x" => 4, "y" => 3)=> ArgumentError (wrong number of arguments (given 1, expected 0))sum(x: 4, y: 3)=> 7

ruby 2.7.0

sum("x" => 4, "y" => 3)=> 7sum(x: 4, y: 3)=> 7

Added support for **nil

Ruby 2.7 added support for **nil to explicitly mention if a method doesn'taccept any keyword arguments in method call.

def sum(a, b, **nil)  a + bendsum(2, 3, x: 4)=> ArgumentError (no keywords accepted)

To suppress above deprecation warnings we can use-W:no-deprecated option.

In conclusion, Ruby 2.7 has worked big steps towards changing specification ofkeyword arguments which will be completely changed in Ruby 3.

For more information on discussion, code changes and official documentation,please head to Feature #14183discussion, pull request andNEWS release.

By applyingEnumerable#lazymethod on any enumerable object, we can convert that object intoEnumerator::Lazy object. The chains of actions on this lazy enumerator will beevaluated only when it is needed. It helps us in processing operations on largecollections, files and infinite sequences seamlessly.

# This line of code will hang and you will have to quit the console by Ctrl+C.irb> list = (1..Float::INFINITY).select { |i| i%3 == 0 }.reject(&:even?)# Just adding `lazy`, the above line of code now executes properly# and returns result without going to infinite loop. Here the chains of# operations are performed as and when it is needed.irb> lazy_list = (1..Float::INFINITY).lazy.select { |i| i%3 == 0 }.reject(&:even?)=> #<Enumerator::Lazy: ...>irb> lazy_list.first(5)=> [3, 9, 15, 21, 27]

When we chain more operations on Enumerable#lazy object, it again returns lazyobject without executing it. So, when we pass lazy objects to any method whichexpects a normal enumerable object as an argument, we have to force evaluationon lazy object by callingto_amethod or it's aliasforce.

# Define a lazy enumerator object.irb> list = (1..30).lazy.select { |i| i%3 == 0 }.reject(&:even?)=> #<Enumerator::Lazy: #<Enumerator::Lazy: ... 1..30>:select>:reject># The chains of operations will return again a lazy enumerator.irb> result = list.select { |x| x if x <= 15 }=> #<Enumerator::Lazy: #<Enumerator::Lazy: ... 1..30>:select>:reject>:map># It returns error when we call usual array methods on result.irb> result.sampleirb> NoMethodError (undefined method `sample'irb> for #<Enumerator::Lazy:0x00007faab182a5d8>)irb> result.lengthirb> NoMethodError (undefined method `length'irb> for #<Enumerator::Lazy:0x00007faab182a5d8>)# We can call the normal array methods on lazy object after forcing# its actual execution with methods as mentioned above.irb> result.force.sample=> 9irb> result.to_a.length=> 3

TheEnumerable#eagermethod returns a normal enumerator from a lazy enumerator, so that lazyenumerator object can be passed to any methods which expects a normal enumerableobject as an argument. Also, we can call other usual array methods on thecollection to get desired results.

# By adding eager on lazy object, the chains of operations would return# actual result here. If lazy object is passed to any method, the# processed result will be received as an argument.irb> eager_list = (1..30).lazy.select { |i| i%3 == 0 }.reject(&:even?).eager=> #<Enumerator: #<Enumerator::Lazy: ... 1..30>:select>:reject>:each>irb> result = eager_list.select { |x| x if x <= 15 }irb> result.sample=> 9irb> result.length=> 3

The same way, we can use eager method when we pass lazy enumerator as anargument to any method which expects a normal enumerator.

irb> list = (1..10).lazy.select { |i| i%3 == 0 }.reject(&:even?)irb> def display(enum)irb>   enum.map { |x| p x }irb> endirb>  display(list)=> #<Enumerator::Lazy: #<Enumerator::Lazy: ... 1..30>:select>:reject>:map>irb> eager_list = (1..10).lazy.select { |i| i%3 == 0 }.reject(&:even?).eagerirb> display(eager_list)=> 3=> 9

Here's the relevantcommitand feature discussion for thischange.

> (1..10).each { |n| p n * 3 }> { a: [1, 2, 3], b: [2, 4, 6], c: [3, 6, 9] }.each { |_k, v| p v }> [10, 100, 1000].each_with_index { |n, i| p n, i }

Ruby 2.7 introduces a new way to access block parameters. Ruby 2.7 onwards, ifblock parameters are obvious and we wish to not use absurd names like n or ietc, we can use numbered parameters which are available inside a block bydefault.

We can use **1_ for first parameter, **2_ for second parameter and so on.

Here's how Ruby 2.7 provides numbered parameters inside a block. Below shown arethe examples from above, only this time using numbered parameters.

> (1..10).each { p _1 * 3 }> { a: [1, 2, 3], b: [2, 4, 6], c: [3, 6, 9] }.each { p _2 }> [10, 100, 1000].each_with_index { p _1, _2 }

Like mentioned inNews-2.7.0 docs,Ruby now raises a warning if we try to define local variable in the format _1.Local variable will have precedence over numbered parameter inside the block.

> _1 = 0> => warning: `_1' is reserved for numbered parameter; consider another name> [10].each { p _1 }> => 0

Numbered parameters are not accessible inside the block if we define ordinaryparameters. If we try to access _1 when ordinary parameters are defined, thenruby raises SyntaxError like shown below.

> ["a", "b", "c"].each_with_index { |alphabet, index| p _1, _2}=> SyntaxError ((irb):1: ordinary parameter is defined)

This feature was suggested 9 years back and came back in discussion last year.After many suggestions community agreed to use _1 syntax.

Head to following links to read the discussion behind numbered parameters,Feature #4475 andDiscussion #15723.

Here's relevantcommitfor this feature.

Before Ruby 2.7, we could have achieved it using group_by or inject.

irb> scores = [100, 35, 70, 100, 70, 30, 35, 100, 45, 30]# we can use group_by to group the scoresirb> scores.group_by { |v| v }.map { |k, v| [k, v.size] }.to_h=> {100=>3, 35=>2, 70=>2, 30=>2, 45=>1}# or we can use inject to group the scoresirb> scores.inject(Hash.new(0)) {|hash, score| hash[score] += 1; hash }=> {100=>3, 35=>2, 70=>2, 30=>2, 45=>1}

Ruby 2.7

Ruby 2.7 adds Enumerable#tally which can be used to find the frequency.Tally makes the code more readable and intuitive. It returns a hash where keysare the unique elements and values are its corresponding frequency.

irb> scores = [100, 35, 70, 100, 70, 30, 35, 100, 45, 30]irb> scores.tally=> {100=>3, 35=>2, 70=>2, 30=>2, 45=>1}

Check out thegithub commitfor more details on this.

JIT stands for Just-In-Time compiler. JIT converts repetitive code into bytecodewhich can then be sent to the processor directly, hence, saving time by notcompiling the same piece of code over and over.

Ruby 2.6

MJIT is introduced in Ruby 2.6. It is most commonly known as MRI JIT or MethodBased JIT.

It is a part of the Ruby 3x3 project started by Matz. The name "Ruby 3x3"signifies Ruby 3.0 will be 3 times faster than Ruby 2.0 and it will focus mainlyon performance. In addition to performance, it also aims for the followingthings:

1. Portability
2. Stability
3. Security

MJIT is still in development, therefore, MJIT is optional in Ruby 2.6. If youare running Ruby 2.6, then you can execute the following command.

ruby --help

You will see following options.

--Jit-wait # Wait program execution until code compiles.--jit-verbose=num # Level information MJIT compiler prints for Ruby program.--jit-min-calls=num # Minimum count in loops for which MJIT should work.--jit-max-cache--jit-save-temps # Save compiled library to the file.

Vladimir Makarov proposed improving performance by replacing VM instructionswith RTL(Register Transfer Language) and introducing the Method based JITcompiler.

Vladimir explained MJIT architecture in hisRubyKaigi 2017 conference keynote.

Ruby's compiler converts the code to YARV(Yet Another Ruby VM) instructions andthen these instructions are run by the Ruby Virtual Machine. Code that isexecuted too often is converted to RTL instructions, which runs faster.

Let's take a look at how MJIT works.

# mjit.rbrequire 'benchmark'puts Benchmark.measure {def test_whilestart_time = Time.nowi = 0    while i < 4      i += 1    end    i    puts Time.now - start_timeend4.times { test_while }}

Let's run this code with MJIT options and check what we got.

ruby --jit --jit-verbose=1 --jit-wait --disable-gems mjit.rb

Time taken is 4.0e-06Time taken is 0.0Time taken is 0.0Time taken is 0.00.000082 0.000032 0.000114 ( 0.000105)Successful MJIT finish

Nothing interesting right? And why is that? because we are iterating the loopfor 4 times and default value for MJIT to work is 5. We can always decide afterhow many calls MJIT should work by providing --jit-min-calls=#number option.

Let's tweak the program a bit so MJIT gets to work.

require 'benchmark'puts Benchmark.measure {def test_whilestart_time = Time.nowi = 0    while i < 4_00_00_000      i += 1    end    puts "Time taken is #{Time.now - start_time}"end10.times { test_while }}

After running the above code we can see some work done by MJIT.

Time taken is 0.457916Time taken is 0.455921Time taken is 0.454672Time taken is 0.452823JIT success (72.5ms): block (2 levels) in <main>@mjit.rb:15 -> /var/folders/v6/\_6sh53vn5gl3lct18w533gr80000gn/T//\_ruby_mjit_p66220u0.cJIT success (140.9ms): test_while@mjit.rb:4 -> /var/folders/v6/\_6sh53vn5gl3lct18w533gr80000gn/T//\_ruby_mjit_p66220u1.cJIT compaction (23.0ms): Compacted 2 methods -> /var/folders/v6/\_6sh53vn5gl3lct18w533gr80000gn/T//\_ruby_mjit_p66220u2.bundleTime taken is 0.463703Time taken is 0.102852Time taken is 0.103335Time taken is 0.103299Time taken is 0.103252Time taken is 0.1032612.797843 0.005357 3.141944 ( 2.801391)Successful MJIT finish

Here's what's happening. Method ran 4 times and on the 5th call it found it isrunning same code again. So MJIT started a separate thread to convert the codeinto RTL instructions, which created a shared object library. Next, threads tookthat shared code and executed directly. As we passed option --jit-verbose=1 wecan see what MJIT did.

What we are seeing in output is the following:

1. Time taken to compile.
2. What block of code is compiled by JIT.
3. Location of compiled code.

We can open the file and see how MJIT converted the piece of code to binaryinstructions but for that we need to pass another option which is--jit-save-temps and then just inspect those files.

After compiling the code to RTL instructions, take a look at the execution time.It dropped down to 0.10 ms from 0.46 ms. That's a neat speed bump.

Here is a comparison across some of the Ruby versions for some basic operations.


Rails comparison on Ruby 2.5, Ruby 2.6 and Ruby 2.6 with JIT

Create a rails application with different Ruby versions and start a server. Wecan start the rails server with the JIT option, as shown below.

RUBYOPT="--jit" bundle exec rails s

Now, we can start testing the performance on servers. We found that Ruby 2.6 isfaster than Ruby 2.5, but enabling JIT in Ruby 2.6 does not add more value tothe Rails application.

MJIT status and future directions

• It is in an early development stage.
• Does not work on windows.
• Needs more time to mature.
• Needs more optimisations.
• MJIT can use GCC or LLVM in the future C Compilers.

Further reading

1. Ruby 3x3 Performance Goal
2. The method JIT compiler for Ruby2.6
3. Vladimir Makarov's Ruby Edition

(1..10).cover?(5)=> true

Range#cover? returns false if the object passed as an argument isnon-comparable or is not in the range.

Before Ruby 2.6, Range#cover? used to return false if a Range object is passedas an argument.

>> (1..10).cover?(2..5)=> false

Ruby 2.6

In Ruby 2.6 Range#cover? can accept a Range object as an argument. It returnstrue if the argument Range is equal to or a subset of the Range.

(1..100).cover?(10..20)=> true(1..10).cover?(2..5)=> true(5..).cover?(4..)=> false("a".."d").cover?("x".."z")=> false

Here is relevantcommitand discussion for this change.

As of now, RubyVM::AST supports two methods: parse and parse_file.

parse method takes a string as a parameter and returns the root node of thetree in the form of the object, RubyVM::AST::Node (Link is not available).

parse_file method takes the file name as a parameter and returns the root nodeof the tree in the form of the object,RubyVM::AST::Node.

Ruby 2.6.0-preview2

irb> RubyVM::AST.parse("(1..100).select { |num| num % 5 == 0 }")=> #<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:38): >irb> RubyVM::AST.parse_file("/Users/amit/app.rb")=> #<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:38): >

RubyVM::AST::Nodehas seven public instance methods - children, first_column, first_lineno,inspect, last_column, last_lineno and type.

Ruby 2.6.0-preview2

irb> ast_node = RubyVM::AST.parse("(1..100).select { |num| num % 5 == 0 }")=> #<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:38): >irb> ast_node.children=> [nil, #<RubyVM::AST::Node(NODE_ITER(9) 1:0, 1:38): >]irb> ast_node.first_column=> 0irb> ast_node.first_lineno=> 1irb> ast_node.inspect=> "#<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:38): >"irb> ast_node.last_column=> 38irb> ast_node.last_lineno=> 1irb> ast_node.type=> "NODE_SCOPE"

This module will majorly help in building a static code analyzer and formatter.

Recently, we saw following error while processing a gif image.

convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.convert-im6.q16: cache resources exhausted `file.gif' @ error/cache.c/OpenPixelCache/3945.

This happens when ImageMagick cannot allocate enough memory to process theimage. This can be fixed by tweaking memory configuration in policy.xml.

Path of policy.xml can be located as follows.

$ identify -list policyPath: /etc/ImageMagick-6/policy.xml  Policy: Resource    name: disk    value: 1GiB

Memory limit can be configured in the following line of policy.xml.

<policy domain="resource" name="memory" value="256MiB"/>

Increasing this value would solve the error if you have a machine with larger amemory.

Let's take an example in which we have to select/filter all numbers which aredivisible by 5 from a range.

Ruby 2.5

irb> (1..100).select { |num| num % 5 == 0 }=> [5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]irb> (1..100).filter { |num| num % 5 == 0 }=> Traceback (most recent call last):2: from /Users/amit/.rvm/rubies/ruby-2.5.1/bin/irb:11:in `<main>' 1: from (irb):2 NoMethodError (undefined method`filter' for 1..100:Range)

Ruby 2.6.0-preview2

irb> (1..100).select { |num| num % 5 == 0 }=> [5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]irb> (1..100).filter { |num| num % 5 == 0 }=> [5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]

Also note, along with Enumerable#filter, Enumerable#filter!,Enumerable#select! was also added as an alias.

Here is the relevant commitand discussion.

Below code will raise class/module name must be CONSTANT exception.

  class   end

We can use above non-ASCII character as a method name or variable name though.

Below code will run without any exception

  class NonAsciiMethodAndVariable    def        = "BigBinary"    end  end

"" is treated as a variable name in above example, even though firstletter() is a capital non-ASCII character.

Ruby 2.6

Ruby 2.6 relaxes above mentioned limitation. We can now define constants inlanguages other than English. Languages having capital letters like Russian andGreek can be used to define constant name.

Below code will run without exception in any Ruby 2.6.

  class   end

As capital non-Ascii characters are now treated as constant, below code willraise a warning in Ruby 2.6.

  irb(main):001:0>  = "BigBinary"  => "BigBinary"  irb(main):002:0>  = "BigBinary"  (irb):2: warning: already initialized constant   (irb):1: warning: previous definition of  was here

Above code will run without any warnings on Ruby versions prior to 2.6

Here is relevant commit anddiscussion for this change.

An alternative Geocoder

Our search for an alternative geocoder landed us on Nominatim. Written in C,with a PHP web interface, Nominatim was performant enough for our requirements.Once set up, Nominatim required 8GB of RAM to run and this included RAM for thePostgreSQL (+ PostGIS) as well.

The rest of the blog discusses how to setup Nominatim and the tips and tricksthat we learned along the way and how it compares with the geocoding solutionoffered by Google.

Setting up Nominatim

We started off by looking for Amazon Machine Images with Nominatim setup andcould only find one which was hosted byOpenStreetMap but the magnet link was dead.

Next, we went through theofficial installation document.We decided to give docker a shot and found that there are many Nominatim dockerbuilds. We usedhttps://github.com/merlinnot/nominatim-dockersince it seemed to follow all the steps mentioned in the official installationguide.

Issues faced during Setup

Out of Memory Errors

The official documentation recommends using 32GB of RAM for initial import butwe needed to double the memory to 64GB to make it work.

Also any time docker build failed, due to the large amount of data that isgenerated on each run, we also ran out of disk space on subsequent docker buildssince docker caches layers across builds.

Merging Multiple Regions

We wanted to geocode locations from USA, Mexico, Canada and Sri Lanka. USA,Mexico and Canada areincluded by default in North America data extractbut we had to merge data for Sri Lanka with North America to get it in a formatrequired for initial import.

The following snippet pre-processes map data for North America and Sri Lankainto a single data.osm.pbf file that can be directly used by Nominatiminstaller.

RUN curl -L 'http://download.geofabrik.de/north-america-latest.osm.pbf' \    --create-dirs -o /srv/nominatim/src/north-america-latest.osm.pbfRUN curl -L 'http://download.geofabrik.de/asia/sri-lanka-latest.osm.pbf' \    --create-dirs -o /srv/nominatim/src/sri-lanka-latest.osm.pbfRUN osmconvert /srv/nominatim/src/north-america-latest.osm.pbf \    -o=/srv/nominatim/src/north-america-latest.o5mRUN osmconvert /srv/nominatim/src/sri-lanka-latest.osm.pbf \    -o=/srv/nominatim/src/sri-lanka-latest.o5mRUN osmconvert /srv/nominatim/src/north-america-latest.o5m \    /srv/nominatim/src/sri-lanka-latest.o5m \    -o=/srv/nominatim/src/data.o5mRUN osmconvert /srv/nominatim/src/data.o5m \    -o=/srv/nominatim/src/data.osm.pbf

Slow Search times

Once the installation was done, we tried running simple locationsearches like this one,but the search timed out. Usually Nominatim can provide a lot of informationfrom its web-interface by appending &debug=true to the search query.

# fromhttps://nominatim.openstreetmap.org/search.php?q=New+York&polygon_geojson=1&viewbox=# tohttps://nominatim.openstreetmap.org/search.php?q=New+York&polygon_geojson=1&viewbox=&debug=true

We created anissue in Nominatim repositoryand got very prompt replies from Nominatim maintainers, especially fromSarah Hoffman .

# runs analyze on the entire nominatim databasepsql -d nominatim -c 'ANALYZE VERBOSE'

PostgreSQL query plannerdepends on statisticscollected bypostgres statistics collectorwhile executing a query. In our case, query planner took an enormous amount oftime to plan queries as there were no stats collected since we had a freshinstallation.

Comparing Nominatim and Google Geocoder

We compared 2500 addresses and we found that Google geocoded 99% of thoseaddresses. In comparison Nominatim could only geocode 47% of the addresses.

It means we still need to geocode ~50% of addresses using Google geocoder. Wefound that we could increase geocoding efficiency by normalizing the addresseswe had.

Address Normalization using libpostal

Libpostal is an address normalizer,which usesstatistical natural-language processingto normalize addresses. Libpostal also hasruby bindings which made it quiteeasy to use it for our test purposes.

Once libpostal and its ruby-bindings were installed (installation isstraightforward and steps are available inruby-postal's github page), we gavelibpostal + Nominatim a go.

require 'geocoder'require 'ruby_postal/expand'require 'ruby_postal/parser'Geocoder.configure({lookup: :nominatim, nominatim: { host: "nominatim_host:port"}})full_address = [... address for normalization ...]expanded_addresses = Postal::Expand.expand_address(full_address)parsed_addresses = expanded_addresses.map do |address|  Postal::Parser.parse_address(address)endparsed_addresses.each do | address |  parsed_address = [:house_number, :road, :city, :state, :postcode, :country].inject([]) do |acc, key|    # address is of format    # [{label: 'postcode', value: 12345}, {label: 'city', value: 'NY'} .. ]    key_value = address.detect { |address| address[:label] == key }    if key_value        acc << "#{key_value_pair[:value]}".titleize    end    acc  end  coordinates = Geocoder.coordinates(parsed_address.join(", "))  if (coordinates.is_a? Array) && coordinates.present?    puts "By Libpostal #{coordinates} => #{parsed_address.join(", ")}"    break  endend

With this, we were able to improve our geocoding efficiency by 10% asNominatim + Libpostal combination could geocode ~ 59% of addresses.

Similarly, write_timeout attribute and write_timeout= method is added toNet::BufferedIO class.

If any chunk of response is not written within number of seconds provided towrite_timeout, Net::WriteTimeoutexception is raised. Net::WriteTimeout exception is not raised on Windowssystems.

Example

# server.rbrequire 'socket'server = TCPServer.new('localhost', 2345)loop dosocket = server.acceptend

Ruby 2.5.1

# client.rbrequire 'net/http'connection = Net::HTTP.new('localhost', 2345)connection.open_timeout = 1connection.read_timeout = 3connection.startpost = Net::HTTP::Post.new('/')body = (('a' _ 1023) + "\n") _ 5_000post.body = bodyputs "Sending #{body.bytesize} bytes"connection.request(post)

Output

\$ RBENV_VERSION=2.5.1 ruby client.rbSending 5120000 bytes

Ruby 2.5.1 processes request endlessly unless above program is interrupted.

Ruby 2.6.0-dev

Add write_timeout attribute to Net::HTTP instance in client.rb program.

# client.rbrequire 'net/http'connection = Net::HTTP.new('localhost', 2345)connection.open_timeout = 1connection.read_timeout = 3# set write_timeout to 10 secondsconnection.write_timeout = 10connection.startpost = Net::HTTP::Post.new('/')body = (('a' _ 1023) + "\n") _ 5_000post.body = bodyputs "Sending #{body.bytesize} bytes"connection.request(post)

Output

\$ RBENV_VERSION=2.6.0-dev ruby client.rbSending 5120000 bytesTraceback (most recent call last):13: `from client.rb:17:in `<main>`` 12: `from /net/http.rb:1479:in `request``11: `from /net/http.rb:1506:in `transport_request`` 10: `from /net/http.rb:1506:in `catch``9: `from /net/http.rb:1507:in `block in transport_request`` 8: `from /net/http/generic_request.rb:123:in `exec``7: `from /net/http/generic_request.rb:189:in `send_request_with_body`` 6: `from /net/protocol.rb:221:in `write``5: `from /net/protocol.rb:239:in `writing`` 4: `from /net/protocol.rb:222:in `block in write``3: `from /net/protocol.rb:249:in `write0`` 2: `from /net/protocol.rb:249:in `each_with_index``1: `from /net/protocol.rb:249:in `each`` `/net/protocol.rb:270:in `block in write0`: Net::WriteTimeout (Net::WriteTimeout)`

In Ruby 2.6.0, above program is terminated raising Net::WriteTimeout exceptionafter 10 seconds (value set to write_timeout attribute).

Here is relevant commit anddiscussion for this change.

In Ruby 2.6, same methods are added as instance methods on Dir class.Dir#children (Link is not available) returns array of all the filenames except. and .. in the directory. Dir#each_child (Link is not available) yields allthe filenames and operates on it.

Let's have a look at examples to understand it better.

Dir#children

directory = Dir.new('/Users/tejaswinichile/workspace')directory.children=> ["panda.png", "apple.png", "banana.png", "camera.jpg"]

Dir#each_child iterates and calls block for each file entry in the givendirectory. It uses filename as a parameter to the block.

Dir#each_child

directory = Dir.new('/Users/tejaswinichile/workspace')directory.each_child { |filename| puts "Currently reading: #{filename}"}Currently reading: panda.pngCurrently reading: apple.pngCurrently reading: banana.pngCurrently reading: camera.jpg=> #<Dir:/Users/tejaswinichile/Desktop>

If we don't pass any block to each_child, it returns enumerator instead.

directory = Dir.new('/Users/tejaswinichile/workspace')directory.each_child=> #<Enumerator: #<Dir:/Users/tejaswinichile/Desktop>:each_child>

Here is relevantcommitand discussion for this change.

> > "1one".to_i> > => 1> > Integer("1one")> > ArgumentError: invalid value for Integer(): "1one"    from (irb):2:in `Integer'    from (irb):2    from /Users/prathamesh/.rbenv/versions/2.4.0/bin/irb:11:in `<main>'> >

The to_i method tries to convert the given input to integer as much aspossible whereas the Integer method throws an ArgumentError if it can'tcovert the input to integer. The Integer and Float methods parse morestrictly compared to to_i and to_f respectively.

Some times, we might need the strictness of Integer and Float but ability tonot raise an exception every time the input can't be parsed.

Before Ruby 2.6 it was possible to achieve it in following way.

> > Integer("msg") rescue nil> >

In Ruby 2.6, theInteger and Float methods accept a keyword argument exceptionwhich can be either true or false. If it is false then no exception israised if the input can't be parsed and nil is returned.

> > Float("foo", exception: false)> > => nil> > Integer("foo", exception: false)> > => nil> >

This is also faster than rescuing the exception and returning nil.

> > Benchmark.ips do |x|> > ?> x.report("rescue") {> > ?> Integer('foo') rescue nil> > }> > x.report("kwarg") {> > ?> Integer('foo', exception: false)> > }> > x.compare!> > end> > Warming up --------------------------------------              rescue    41.896k i/100ms               kwarg    81.459k i/100msCalculating -------------------------------------rescue 488.006k ( 4.5%) i/s - 2.472M in 5.076848skwarg 1.024M (11.8%) i/s - 5.050M in 5.024937sComparison:kwarg: 1023555.3 i/srescue: 488006.0 i/s - 2.10x slower

As we can see, rescuing the exception is twice slower than using the new keywordargument. We can still use the older technique if we want to return a differentvalue from nil.

> > Integer('foo') rescue 42> > => 42> >

By default, the keyword argument exception is set to true for backwardcompatibility.

The Chinese version of this blog is availablehere.

binding.eval('[__FILE__, __LINE__]')=> ["/Users/taha/blog/app/controllers/application_controller", 2]

Ruby 2.6 adds more readable method Binding#source_location (Link is notavailable) to achieve similar result.

binding.source_location=> ["/Users/taha/blog/app/controllers/application_controller", 2]

Here is relevant commit anddiscussion for this change.

The Chinese version of this blog is availablehere.

In Ruby 2.6, a block can be passed to String#split (Link is not available) whichyields each split string and operates on it. This avoids creating an array andthus is memory efficient.

We will add method is_fruit? to understand how to use split with a block.

def is_fruit?(value)%w(apple mango banana watermelon grapes guava lychee).include?(value)end

Input is a comma separated string with vegetables and fruits names. Goal is tofetch names of fruits from input string and store it in an array.

String#split

input_str = "apple, mango, potato, banana, cabbage, watermelon, grapes"splitted_values = input_str.split(", ")=> ["apple", "mango", "potato", "banana", "cabbage", "watermelon", "grapes"]fruits = splitted_values.select { |value| is_fruit?(value) }=> ["apple", "mango", "banana", "watermelon", "grapes"]

Using split an intermediate array is created which contains both fruits andvegetables names.

String#split with a block

fruits = []input_str = "apple, mango, potato, banana, cabbage, watermelon, grapes"input_str.split(", ") { |value| fruits << value if is_fruit?(value) }=> "apple, mango, potato, banana, cabbage, watermelon, grapes"fruits=> ["apple", "mango", "banana", "watermelon", "grapes"]

When a block is passed to split, it returns the string on which split wascalled and does not create an array. String#split yields block on each splitstring, which in our case was to push fruit names in a separate array.

Update

Benchmark

We created a large random string to benchmark performance of split andsplit with block

require 'securerandom'test_string = ''100_000.times.each dotest_string += SecureRandom.alphanumeric(10)test_string += ' 'end

require 'benchmark'Benchmark.bmbm do |bench|bench.report('split') doarr = test_string.split(' ')str_starts_with_a = arr.select { |str| str.start_with?('a') }endbench.report('split with block') dostr_starts_with_a = []test_string.split(' ') { |str| str_starts_with_a << str if str.start_with?('a') }endend

Results

Rehearsal ----------------------------------------------------split              0.023764   0.000911   0.024675 (  0.024686)split with block   0.012892   0.000553   0.013445 (  0.013486)------------------------------------------- total: 0.038120sec                       user     system      total        realsplit              0.024107   0.000487   0.024594 (  0.024622)split with block   0.010613   0.000334   0.010947 (  0.010991)

We did another iteration of benchmarking usingbenchmark/ips.

require 'benchmark/ips'Benchmark.ips do |bench|bench.report('split') dosplitted_arr = test_string.split(' ')str_starts_with_a = splitted_arr.select { |str| str.start_with?('a') }endbench.report('split with block') dostr_starts_with_a = []test_string.split(' ') { |str| str_starts_with_a << str if str.start_with?('a') }endbench.compare!end

Results

Warming up --------------------------------------               split     4.000  i/100ms    split with block    10.000  i/100msCalculating -------------------------------------               split     46.906  ( 2.1%) i/s -    236.000  in   5.033343s    split with block    107.301  ( 1.9%) i/s -    540.000  in   5.033614sComparison:    split with block:      107.3 i/s               split:       46.9 i/s - 2.29x  slower

This benchmark shows that split with block is about 2 times faster thansplit.

Here is relevantcommitand discussion for this change.

The Chinese version of this blog is availablehere.

If we use else without rescue inside begin..end block in Ruby 2.5, itgives a warning.

irb(main):001:0> beginirb(main):002:1> puts "Inside begin block"irb(main):003:1> elseirb(main):004:1> puts "Inside else block"irb(main):005:1> end(irb):5: warning: else without rescue is useless

This warning is present as code inside else block will never get executed

Ruby 2.6

In Ruby 2.6 it will raise an exception if we use else without rescue inbegin..end block. Thiscommitchanged warning into exception in Ruby 2.6. Changes made in the commit areexperimental.

irb(main):001:0> beginirb(main):002:1> puts "Inside begin block"irb(main):003:1> elseirb(main):004:1> puts "Inside else block"irb(main):005:1> endTraceback (most recent call last):1: from /usr/local/bin/irb:11:in `<main>'SyntaxError ((irb):3: else without rescue is useless)

The Chinese version of this blog is availablehere.

Ruby 2.5.0

irb> (1..Float::INFINITY).each do |n|irb\* # logic goes hereirb> end

OR

irb> 1.step.each do |n|irb\* # logic goes hereirb> end

Ruby 2.6.0

Ruby 2.6 makes infinite loop more readable by changing mandatory second argumentin range to optional. Internally, Ruby changes second argument to nil ifsecond argument is not provided. So, both (0..) and (0..nil) are same inRuby 2.6.

Using endless loop in Ruby 2.6

irb> (0..).each do |n|irb\* # logic goes hereirb> end

irb> (0..nil).size=> Infinityirb> (0..).size=> Infinity

In Ruby 2.5, nil is not an acceptable argument and (0..nil) would throwArgumentError.

irb> (0..nil)ArgumentError (bad value for range)

Here is the relevantcommitand discussion for this change.

The Chinese version of this blog is availablehere.

Using Concurrent::Future

In one of our applications, to improve performance we added threaded code usingConcurrent::Future. Itworked really well for us until one day it stopped working.

"Why threads?" one might ask. The code in question was a textbook threading usecase. It had a few API calls, some DB requests and finally an action that wasperformed on all the data that was aggregated.

Let us look at what this code looks like.

Non threaded code

selected_shipping_companies.each do | carrier |  # api calls  distance_in_miles = find_distance_from_origin_to_destination  historical_average_rate = historical_average_for_this_particular_carrier  # action performed  build_price_details_for_this_carrier(distance_in_miles,                                       historical_average_rate)end

Converting the above code to use Concurrent::Future is trivial.

futures = selected_shipping_companies.map do |carrier|  Concurrent::Future.execute do    # api calls    distance_in_miles = find_distance_from_origin_to_destination    historical_average_rate = historical_average_for_this_particular_carrier    # action performed    build_price_details_for_this_carrier(distance_in_miles,                                         historical_average_rate)  endendfutures.map(&:value)

A bit more about Concurrent::Future

It is often intimidating to work with threads. They can bring in complexity andcan have unpredictable behaviors due to lack of thread-safety. Ruby, being alanguage of mutable references, we often find it difficult to write 100%thread-safe code.

Inspired byClojure's Future function,Concurrent::Future is a primitive that guarantees thread safety. It takes ablock of work and performs the work asynchronously using Concurrent Ruby'sglobal thread-pool. Once a block of work is scheduled, Concurrent Ruby gives usa handle to this future work, on which when #value (or #deref) is called block'svalue is returned.

The Bug

Usually, when an exception occurs in the main thread, the interpreter stops andgathers the exception data. In the case of Ruby Threads, any unhandledexceptions are reported only whenThread#join iscalled. Setting Thread#abort_on_exception to true, is an better alternativewhich will cause all threads to exit when an exception is raised in any runningthread. Wepublished a blogrecently which talks about this in great detail.

Exception handling in Concurrent Ruby

future = Concurrent::Future.execute {            raise StandardError.new("Boom!")          }sleep(0.1) # giving arbitrary time for future to executefuture.value     #=> nil

Where did the exception go? This code fails silently and swallows theexceptions. How can we find out if the code executed successfully?

future = Concurrent::Future.execute {              raise StandardError.new("Boom!")          }sleep(0.1) # giving arbitrary time for future to executefuture.value     #=> nilfuture.rejected? #=> truefuture.reason    #=> "#<StandardError: Boom!>"

How we fixed our issue

We found places in our application where Concurrent::Future was used in a waythat would swallow exceptions. It is also a possibility that people mightoverlook the explicit need to manually report exception. We addressed theseconcerns with the following wrapper class.

module ConcurrentExecutor  class Error < StandardError    def initialize(exceptions)      @exceptions = exceptions      super    end    def message      @exceptions.map { | e | e.message }.join "\n"    end    def backtrace      traces = @exceptions.map { |e| e.backtrace }      ["ConcurrentExecutor::Error START", traces, "END"].flatten    end  end  class Future    def initialize(pool: nil)      @pool = pool || Concurrent::FixedThreadPool.new(20)      @exceptions = Concurrent::Array.new    end    # Sample Usage    # executor = ConcurrentExecutor::Future.new(pool: pool)    # executor.execute(carriers) do | carrier |    #   ...    # end    #    # values = executor.resolve    def execute array, &block      @futures = array.map do | element |        Concurrent::Future.execute({ executor: @pool }) do          yield(element)        end.rescue do | exception |          @exceptions << exception        end      end      self    end    def resolve      values = @futures.map(&:value)      if @exceptions.length > 0        raise ConcurrentExecutor::Error.new(@exceptions)      end      values    end  endend

Please note that using Concurrent Ruby Futures caused segmentation fault whilerunning specs in Circle CI. As of this writing, we are using normal loopinginstead of Futures in Circle CI until the reason for the segfault is isolatedand fixed.

Update

Concurrent::Future also gives us another API which not only returns the value ofthe block but also posts/raises any exceptions that occur into the main thread.

thread_pool = Concurrent::FixedThreadPool.new(20)executors = [1, 2, 3, 4].map do |random_number|  Concurrent::Future.execute({ executor: thread_pool }) do    random_number / (random_number.even? ? 0 : 1)  endendexecutors.map(&:value)=> [1, nil, 3, nil]executors.map(&:value!)> ZeroDivisionError: divided by 0> from (pry):4:in `/'

We thank Jonathan Rochkind for pointing us tothis undocumented apiin his reddit post.

This is Prathamesh. I bring you live detailsabout what is happening at the Kaigi over the next two days. If you are at theconference please come and say "Hi" to me.

Check outwhat happened on day 1.

Faster Apps, No Memory Thrash: Get Your Memory Config Right by Noah Gibbs

Noah gave an awesome talk on techniques tomanage the memory used by Ruby applications. One of the main point while dealingwith GC is to make it run less, which means don't create too many objects. Healso mentioned that if application permits then destructive operations such asgsub! or concat should be used since they save CPU cycles and memory. Rubyallows setting up environment variables for managing the heap memory but it isreally hard to choose values for these environment variables blindly.

Noah has built a tool which uses GC.stat results from applications to estimatethe values of the memory related environment variables. Check out theEnvMem gem.

In the end, he discussed some advanced debugging methods like checkingfragmentation percentage. The formula is prepared byNate Berkopec.

s = GC.statused_ratio = s[:heap_live_slots].to_f / (s[:heap_eden_pages] * 408)fragmentation = 1 - used_ratio

We can also use GC::Profiler to profile the code in real time to see how GC isbehaving.

Benchmark used for this talk can be foundhere. Slides for this talk canbe foundhere.

Guild prototype

Next I attended talk by Koichi Sasada on Guildprototype. He discussed the proposed design spec for Guild with a demo ofFibonacci number program with 40 core CPU and 40 guilds. One of the interestingobservations is that performance drops as number of guilds increases because ofthe global locking.

He discussed the concept of shareable and non-shareable objects. Shareableobjects can be shared across multiple Guilds whereas non-shareable objects canonly be used one Guild. This also means, you can't make thread unsafe programwith Guilds "by design". He discussed about the challenges in specifying theshareable objects for Guilds.

Overall, there is still a lot of work left to be done for Guilds to become apart of Ruby. It includes defining protocols for shareable and non-shareableobjects, making sure GC runs properly in case of Guilds, synchronization betweendifferent Guilds.

The slides for this talk can be foundhere.

Ruby programming with type checking

Soutaro from SideCI gave a talk onSteep, a gradual type checker for Ruby.

In the past, Matz has said that he doesn't like type definitions to be presentin the Ruby code. Steep requires type definitions to be present in separatefiles with extension .rbi. The Ruby source code needs little amount ofannotations. Steep also has a scaffold generator to generate the basic typedefinitions for existing code.

As of now, Steep runs slower than Sorbet which wasdiscussed yesterday by Stripe team. Soutaro also discussed issues in typedefinitions due to meta programming in libraries such as Active Record. Thatlooks like a challenge for Steep as of now.

Web console

After the tea break, I attended talk by Genadi on howweb console works.

He discussed the implementation of web-console in detail with references to Rubyinternals related to bindings. He compared the web-console interface with IRBand pry and explained the difference. As of now, web console has to monkey patchsome of the Rails internals. Genadi has added support forregistering interceptors which willprevent this monkey patching in Rails 6. He is also mentoring a Google summer ofcode student to work on Actionable errors project where the user can takeactions like running pending migrations via the webpage itself when the error isshown.

Ruby committers v/s the World

RubyKaigi offers this unique event where all the Ruby committers come on stageand face the questions from the audience. This year the format was slightlydifferent and it was run in the style of the Ruby core developer meeting. Theagendawas predecided with some questions from the people and some tickets to discuss.The session started with discussion offeatures coming up in Ruby 2.6.

After that, the questions and the tickets in the agenda were discussed. It wasgood to see how the Ruby team takes decisions about features and suggestions.

Apart from this, there weretalks on SciRuby, mRuby, linting, gem upgrades, c extensions and morewhich I could not attend.

That's all for the day two. Looking forward to the day three already!

Oh and here is the world map of all the attendees from RubyKaigi.

This is Prathamesh.I bring you live details about what is happeningat the Kaigi over the next three days.If you are at the conference please come and say "Hi" to me.

Matz's keynote

RubyKaigi started with Matz's keynote. He used lot of proverbs applyingthem to the Ruby language and software development.

He talked about one of the hardest problems in programming - naming with an example ofyield_self.Matz added alias then to the yield_self method yesterday. He also discussed about googlability of the names.Ironically, Ruby was named in 1993 which was before Google had started.

Matz also touched uponJIT option being introduced in Ruby 2.6and guild as the ways the language continues to improve inperformance and concurrency.There is a talk on Guild by Koichi Sasada on second day of RubyKaigi whichwill have further details about it.

Matz ended the keynote talking about the need of maintaining backward compatibilityand not running into the situation like Ruby 1.9 or Python 3 where the compatibility was notmaintained. He also stressed upon the community aspect of the Ruby language and its importancein the success of Ruby.

ETL processing in Ruby using Kiba

Thibaut Barrregave a talk onKiba - a data processing ETL framework for Ruby.He discussed aboutthe design decisions that went into the version 1 and how it evolved to version 2 which was recently released.

Kiba provides programmatic API which can be used in the background jobs instead of shelling out. It alsohas support for multistep batch processing.

Thibaut also explained how it can be used for data migration,reusing the components and big rewrites. He observed that the performance has been gradually increasing with each Ruby release over the years.

The slides for this talk can be found here.

Architecture of Hanami applications

Next I attended talk fromAnton Davydovon architecture patterns in Hanami apps.He discussed about the problems typical Rails applications faceandhow abstractions can address those issues.He explained how Hanami tries to achieve business logic isolation, avoid global state, sequential logic and test coverage.Functional callable objects, containers, dry-containers, dry-inject and event sourcing are some of the abstractions that can be used in Hanami apps that help inachieving this.

Lightning talks

The last session of the day was lightning talks.

The talk onRib(wordplay on IRB) was an interesting one. Rib is yet another interactive Ruby shell but lightweight compared to IRB and pry. It has some nice features like auto indent, multiline history, filtering of callers. It can also beep when the console starts, so you know it is time to get back to work.

I liked another talk whereWatsonhad worked on improving the performance of JSON gem.He achieved this by using CRuby API wherever applicable and avoiding heavy calls like rbfuncall.Check these twopullrequests for benchmark and more discussions.

Apart from these talks,there were lot of other talks as well which I could not attend.Stripe teamis building atype checker for Rubywhich looks very interesting and is extremely fast.

Bozhidar Batsovgave a talk on Rubocop project andhow it has evolved over the years.There was also a talk on Karafka - event driven architecture in Ruby.This talk was a good precursor to the Hanami talk where event driven architecture was mentioned again.

Other talks from day oneranged from memory management, playing with Ruby syntax, code highlighter, deep learning, C extensions to Rubygems.

That's all for the day one. Looking forward to the day two already!

In Ruby 2.4 when we pass a block to a method, which further passes to anothermethod, Ruby creates a new Proc object by the given block before passing thisproc to the another method.

This creates unnecessary objects even when the block parameter is not accessed.It also creates a chain of Proc objects when the block parameter is passedthrough various methods.

Proc creation is one a heavyweight operation because we need to store all localvariables (represented by Env objects in MRI internal) in the heap.

Ruby 2.5 introduced a lazy proc allocation. Ruby 2.5 will not create a Procobject when passing a block to another method. Instead, it will pass the blockinformation. If the block is accessed somewhere else, then it creates a Procobject by the given block.

This results in lesser memory allocation and faster execution.

Ruby 2.4

irb> require 'benchmark'  => trueirb> def greetirb>   yieldirb> end  => :greetirb>irb> def greet_with_welcome(&block)irb>   puts 'Welcome'irb>   greet(&block)irb> end  => :greet_with_welcomeirb>irb> Benchmark.measure { 1000.times { greet_with_welcome { 'BigBinary' } } }WelcomeWelcome.........  => #<Benchmark::Tms:0x007fe6ab929de0 @label="", @real=0.022295999999187188, @cstime=0.0, @cutime=0.0, @stime=0.01, @utime=0.0, @total=0.01>

Ruby 2.5

irb> require 'benchmark'  => trueirb> def greetirb>   yieldirb> end  => :greetirb>irb> def greet_with_welcome(&block)irb>   puts 'Welcome'irb>   greet(&block)irb> end  => :greet_with_welcomeirb>  irb> Benchmark.measure { 1000.times { greet_with_welcome { 'BigBinary' } } }WelcomeWelcome.........  => #<Benchmark::Tms:0x00007fa4400871b8 @label="", @real=0.004612999997334555, @cstime=0.0, @cutime=0.0, @stime=0.001524000000000001, @utime=0.0030690000000000023, @total=0.004593000000000003>

As we can see, there is considerable improvement in execution time when a blockparam is passed in Ruby 2.5.

Here is the relevant commitand discussion.

OSS is free and open source and has basic features.Pro and Enterprise versions are closed source and paid,thus comes with more advanced features.To compare the list of features offered by each of these versions,please visit Sidekiq website.

Sidekiq Pro 3.4.0introducedsuper_fetch strategyto reliably fetch jobs from the queue in Redis.

In this post, we will discuss the benefits of using super_fetch strategy.

Problem

Open source version of Sidekiq comes with basic_fetch strategy.Let's see an example to understand how it works.

Let's add Sidekiq to our Gemfile and run bundle install to install it.

gem 'sidekiq'

Add following Sidekiq worker in app/workers/sleep_worker.rb.

class SleepWorker  include Sidekiq::Worker  def perform(name)    puts "Started #{name}"    sleep 30    puts "Finished #{name}"  endend

This worker does nothing great but sleeps for 30 seconds.

Let's open Rails consoleand schedule this worker to run as a background job asynchronously.

>> require "sidekiq/api"=> true>> Sidekiq::Queue.new.size=> 0>> SleepWorker.perform_async("A")=> "5d8bf898c36a60a1096cf4d3">> Sidekiq::Queue.new.size=> 1

As we can see, queue now has 1 job scheduled to be processed.

Let's start Sidekiq in another terminal tab.

$ bundle exec sidekiq40510 TID-owu1swr1i INFO: Booting Sidekiq 5.1.3 with redis options {:id=>"Sidekiq-server-PID-40510", :url=>nil}40510 TID-owu1swr1i INFO: Starting processing, hit Ctrl-C to stop40510 TID-owu1tr5my SleepWorker JID-5d8bf898c36a60a1096cf4d3 INFO: startStarted A

As we can see, the job with ID 5d8bf898c36a60a1096cf4d3was picked up by Sidekiqand it started processing the job.

If we check the Sidekiq queue size in the Rails console, it will be zero now.

>> Sidekiq::Queue.new.size=> 0

Let's shutdown the Sidekiq process gracefullywhile Sidekiq is still in the middle of processing our scheduled job.Press either Ctrl-C or run kill -SIGINT <PID> command.

$ kill -SIGINT 40510

40510 TID-owu1swr1i INFO: Shutting down40510 TID-owu1swr1i INFO: Terminating quiet workers40510 TID-owu1x00rm INFO: Scheduler exiting...40510 TID-owu1swr1i INFO: Pausing to allow workers to finish...40510 TID-owu1swr1i WARN: Terminating 1 busy worker threads40510 TID-owu1swr1i WARN: Work still in progress [#<struct Sidekiq::BasicFetch::UnitOfWork queue="queue:default", job="{\"class\":\"SleepWorker\",\"args\":[\"A\"],\"retry\":true,\"queue\":\"default\",\"jid\":\"5d8bf898c36a60a1096cf4d3\",\"created_at\":1525427293.956314,\"enqueued_at\":1525427293.957355}">]40510 TID-owu1swr1i INFO: Pushed 1 jobs back to Redis40510 TID-owu1tr5my SleepWorker JID-5d8bf898c36a60a1096cf4d3 INFO: fail: 19.576 sec40510 TID-owu1swr1i INFO: Bye!

As we can see, Sidekiq pushed back the unfinished job back to Redis queuewhen Sidekiq received a SIGINT signal.

Let's verify it.

>> Sidekiq::Queue.new.size=> 1

Before we move on, let's learn some basics about signals such as SIGINT.

A crash course on POSIX signals

SIGINT is an interrupt signal.It is an alternative to hittingCtrl-C from the keyboard.When a process is running in foreground,we can hit Ctrl-C to signal the process to shut down.When the process is running in background,we can use kill command to send a SIGINT signal to the process' PID.A process can optionally catch this signal and shutdown itself gracefully.If the process does not respect this signal and ignores it,then nothing really happens and the process keeps running.Both INT and SIGINT are identical signals.

Another useful signal is SIGTERM.It is called a termination signal.A process can either catch itand perform necessary cleanup or just ignore it.Similar to a SIGINT signal,if a process ignores this signal, then the process keeps running.Note that, if no signal is supplied to the kill command,SIGTERM is used by default.Both TERM and SIGTERM are identical signals.

SIGTSTP or TSTP is called terminal stop signal.It is an alternative to hitting Ctrl-Z on the keyboard.This signal causes a process to suspend further execution.

SIGKILL is known as kill signal.This signal is intended to kill the process immediately and forcefully.A process cannot catch this signal,therefore the process cannot perform cleanup or graceful shutdown.This signal is usedwhen a process does not respect and respondto both SIGINT and SIGTERM signals.KILL, SIGKILL and 9 are identical signals.

There are a lot of other signals besides these,but they are not relevant for this post.Please check them out here.

A Sidekiq process pays respectto all of these signals and behaves as we expect.When Sidekiq receives a TERM or SIGTERM signal,Sidekiq terminates itself gracefully.

Back to our example

Coming back to our example from above,we had sent a SIGINT signal to the Sidekiq process.

$ kill -SIGINT 40510

On receiving this SIGINT signal,Sidekiq process having PID 40510 terminated quiet workers,paused the queue and waited for a whileto let busy workers finish their jobs.Since our busy SleepWorker did not finish quickly,Sidekiq terminated that busy workerand pushed it back to the queue in Redis.After that, Sidekiq gracefully terminated itself with an exit code 0.Note that, the default timeout is 8 secondsuntil which Sidekiq can wait to let the busy workers finishotherwise it pushes the unfinished jobs back to the queue in Redis.This timeout can be changed with -t optiongiven at the startup of Sidekiq process.

Sidekiq recommendsto send a TSTP and a TERM togetherto ensure that the Sidekiq process shuts down safely and gracefully.On receiving a TSTP signal,Sidekiq stops pulling new workandfinishes the work which is in-progress.The idea is to first send a TSTP signal,wait as much as possible (by default for 8 seconds as discussed above)to ensure that busy workers finish their jobsand then send a TERM signalto shutdown the process.

Sidekiq pushes back the unprocessed job in Redis when terminated gracefully.It means that Sidekiq pulls the unfinished job and starts processing again whenwe restart the Sidekiq process.

$ bundle exec sidekiq45916 TID-ovfq8ll0k INFO: Booting Sidekiq 5.1.3 with redis options {:id=>"Sidekiq-server-PID-45916", :url=>nil}45916 TID-ovfq8ll0k INFO: Starting processing, hit Ctrl-C to stop45916 TID-ovfqajol4 SleepWorker JID-5d8bf898c36a60a1096cf4d3 INFO: startStarted AFinished A45916 TID-ovfqajol4 SleepWorker JID-5d8bf898c36a60a1096cf4d3 INFO: done: 30.015 sec

We can see that Sidekiq pulled the previously terminated jobwith ID 5d8bf898c36a60a1096cf4d3 and processed that job again.

So far so good.

This behavior is implemented usingbasic_fetchstrategy which is present in the open sourced version of Sidekiq.

Sidekiq uses BRPOP Redis commandto fetch a scheduled job from the queue.When a job is fetched,that job gets removed from the queue andthat job no longer exists in Redis.If this fetched job is processed, then all is good.Also, if the Sidekiq process is terminated gracefully onreceiving either a SIGINT or a SIGTERM signal,Sidekiq will push back the unfinished jobs back to the queue in Redis.

But what if the Sidekiq process crashes in the middlewhile processing that fetched job?

A process is considered as crashedif the process does not shutdown gracefully.As we discussed before,when we send a SIGKILL signal to a process,the process cannot receive or catch this signal.Because the process cannot shutdown gracefully and nicely,it gets crashed.

When a Sidekiq process is crashed,the fetched jobs by that Sidekiq processwhich are not yet finished get lostforever.

Let's try to reproduce this scenario.

We will schedule another job.

>> SleepWorker.perform_async("B")=> "37a5ab4139796c4b9dc1ea6d">> Sidekiq::Queue.new.size=> 1

Now, let's start Sidekiq process and kill it using SIGKILL or 9 signal.

$ bundle exec sidekiq47395 TID-ow8q4nxzf INFO: Starting processing, hit Ctrl-C to stop47395 TID-ow8qba0x7 SleepWorker JID-37a5ab4139796c4b9dc1ea6d INFO: startStarted B[1]    47395 killed     bundle exec sidekiq

$ kill -SIGKILL 47395

Let's check if Sidekiq had pushed the busy (unprocessed) jobback to the queue in Redis before terminating.

>> Sidekiq::Queue.new.size=> 0

No. It does not.

Actually, the Sidekiq process did not get a chance to shutdown gracefullywhen it received the SIGKILL signal.

If we restart the Sidekiq process,it cannot fetch that unprocessed jobsince the job was not pushed back to the queue in Redis at all.

$ bundle exec sidekiq47733 TID-ox1lau26l INFO: Booting Sidekiq 5.1.3 with redis options {:id=>"Sidekiq-server-PID-47733", :url=>nil}47733 TID-ox1lau26l INFO: Starting processing, hit Ctrl-C to stop

Therefore,the job having name argument as B or ID as 37a5ab4139796c4b9dc1ea6dis completely lost.There is no way to get that job back.

Losing job like this may not be a problem for some applicationsbut for some critical applications this could be a huge issue.

We faced a similar problem like this.One of our clients' application is deployed on a Kubernetes cluster.Our Sidekiq process runs in a Docker containerin the Kubernetespodswhich we call background pods.

Here's our stripped down version ofKubernetes deploymentmanifest which creates a Kubernetes deployment resource.Our Sidekiq process runs in the pods spawned by that deployment resource.

---apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: backgroundspec:  replicas: 2  template:    spec:      terminationGracePeriodSeconds: 60      containers:      - name: background        image: <%= ENV['IMAGE'] %>        env:        - name: POD_TYPE          value: background        lifecycle:          preStop:            exec:              command:              - /bin/bash              - -l              - -c              - for pid in tmp/pids/sidekiq*.pid; do bin/bundle exec sidekiqctl stop $pid 60; done

When we apply an updated version of this manifest,for say, changing the Docker image, the running pods are terminatedand new pods are created.

Before terminating the only container in the pod,Kubernetes executes sidekiqctl stop $pid 60 commandwhich we have defined using thepreStopevent handler.Note that, Kubernetes already sends SIGTERM signalto the container being terminated inside the podbefore invoking the preStop event handler.The default termination grace period is 30 seconds and it is configurable.If the container doesn't terminate within the termination grace period,a SIGKILL signal will be sent to forcefully terminate the container.

The sidekiqctl stop $pid 60 command executed in the preStop handler doesthree things.

1. Sends a SIGTERM signal to the Sidekiq process running in the container.
2. Waits for 60 seconds.
3. Sends a SIGKILL signal to kill the Sidekiq process forcefullyif the process has not terminated gracefully yet.

This worked for us when the count of busy jobs was relatively small.

When the number of processing jobs is higher,Sidekiq does not get enough timeto quiet the busy workersand fails to push some of them back on the Redis queue.

We found that some of the jobs were getting lostwhen our background pod restarted.We had to restart our background pod forreasons such asupdating the Kubernetes deployment manifest,pod being automatically evicted by Kubernetesdue to host node encountering OOM (out of memory) issue, etc.

We tried increasing bothterminationGracePeriodSeconds in the deployment manifestas well as the sidekiqctl stop command's timeout.Despite that,we still kept facing the same issueof losing jobs whenever pod restarts.

We even tried sending TSTP and then TERM after a timeoutrelatively longer than 60 seconds.But the pod was getting harshly terminatedwithout gracefully terminating Sidekiq process running inside it.Therefore we kept losing the busy jobswhich were running during the pod termination.

Sidekiq Pro's super_fetch

We were looking for a way to stop losing our Sidekiq jobsor a way to recover them reliably when our background Kubernetes pod restarts.

We realized that the commercial version of Sidekiq,Sidekiq Pro offers an additional fetch strategy,super_fetch,which seemed more efficient and reliablecompared to basic_fetch strategy.

Let's see what difference super_fetch strategymakes over basic_fetch.

We will need to use sidekiq-pro gem which needs to be purchased.Since Sidekiq Pro gem is close sourced, we cannot fetch itfrom the default public gem registry,https://rubygems.org.Instead, we will have to fetch it from a private gem registrywhich we get after purchasing it.We add following code to our Gemfile and run bundle install.

source ENV['SIDEKIQ_PRO_GEM_URL'] do  gem 'sidekiq-pro'end

To enable super_fetch,we need to add following codein an initializer config/initializers/sidekiq.rb.

Sidekiq.configure_server do |config|  config.super_fetch!end

Well, that's it.Sidekiq will use super_fetch instead of basic_fetch now.

$ bundle exec sidekiq75595 TID-owsytgvqj INFO: Sidekiq Pro 4.0.2, commercially licensed.  Thanks for your support!75595 TID-owsytgvqj INFO: Booting Sidekiq 5.1.3 with redis options {:id=>"Sidekiq-server-PID-75595", :url=>nil}75595 TID-owsytgvqj INFO: Starting processing, hit Ctrl-C to stop75595 TID-owsys5imz INFO: SuperFetch activated

When super_fetch is activated, Sidekiq process' graceful shutdown behavioris similar to that of basic_fetch.

>> SleepWorker.perform_async("C")=> "f002a41393f9a79a4366d2b5">> Sidekiq::Queue.new.size=> 1

$ bundle exec sidekiq76021 TID-ow6kdcca5 INFO: Sidekiq Pro 4.0.2, commercially licensed.  Thanks for your support!76021 TID-ow6kdcca5 INFO: Booting Sidekiq 5.1.3 with redis options {:id=>"Sidekiq-server-PID-76021", :url=>nil}76021 TID-ow6kdcca5 INFO: Starting processing, hit Ctrl-C to stop76021 TID-ow6klq2cx INFO: SuperFetch activated76021 TID-ow6kiesnp SleepWorker JID-f002a41393f9a79a4366d2b5 INFO: startStarted C

>> Sidekiq::Queue.new.size=> 0

$ kill -SIGTERM 76021

76021 TID-ow6kdcca5 INFO: Shutting down76021 TID-ow6kdcca5 INFO: Terminating quiet workers76021 TID-ow6kieuwh INFO: Scheduler exiting...76021 TID-ow6kdcca5 INFO: Pausing to allow workers to finish...76021 TID-ow6kdcca5 WARN: Terminating 1 busy worker threads76021 TID-ow6kdcca5 WARN: Work still in progress [#<struct Sidekiq::Pro::SuperFetch::Retriever::UnitOfWork queue="queue:default", job="{\"class\":\"SleepWorker\",\"args\":[\"C\"],\"retry\":true,\"queue\":\"default\",\"jid\":\"f002a41393f9a79a4366d2b5\",\"created_at\":1525500653.404454,\"enqueued_at\":1525500653.404501}", local_queue="queue:sq|vishal.local:76021:3e64c4b08393|default">]76021 TID-ow6kdcca5 INFO: SuperFetch: Moving job from queue:sq|vishal.local:76021:3e64c4b08393|default back to queue:default76021 TID-ow6kiesnp SleepWorker JID-f002a41393f9a79a4366d2b5 INFO: fail: 13.758 sec76021 TID-ow6kdcca5 INFO: Bye!

>> Sidekiq::Queue.new.size=> 1

That looks good.As we can see, Sidekiq moved busy job back from a private queueto the queue in Rediswhen Sidekiq received a SIGTERM signal.

Now, let's try to kill Sidekiq process forcefullywithout allowing a graceful shutdownby sending a SIGKILL signal.

Since Sidekiq was gracefully shutdown before,if we restart Sidekiq again,it will re-process the pushed back job having ID f002a41393f9a79a4366d2b5.

$ bundle exec sidekiq76890 TID-oxecurbtu INFO: Sidekiq Pro 4.0.2, commercially licensed.  Thanks for your support!76890 TID-oxecurbtu INFO: Booting Sidekiq 5.1.3 with redis options {:id=>"Sidekiq-server-PID-76890", :url=>nil}76890 TID-oxecurbtu INFO: Starting processing, hit Ctrl-C to stop76890 TID-oxecyhftq INFO: SuperFetch activated76890 TID-oxecyotvm SleepWorker JID-f002a41393f9a79a4366d2b5 INFO: startStarted C[1]    76890 killed     bundle exec sidekiq

$ kill -SIGKILL 76890

>> Sidekiq::Queue.new.size=> 0

It appears that Sidekiq didn't get any chanceto push the busy job back to the queue in Redison receiving a SIGKILL signal.

So, where is the magic of super_fetch?

Did we lose our job again?

Let's restart Sidekiq and see it ourself.

$ bundle exec sidekiq77496 TID-oum04ghgw INFO: Sidekiq Pro 4.0.2, commercially licensed.  Thanks for your support!77496 TID-oum04ghgw INFO: Booting Sidekiq 5.1.3 with redis options {:id=>"Sidekiq-server-PID-77496", :url=>nil}77496 TID-oum04ghgw INFO: Starting processing, hit Ctrl-C to stop77496 TID-oum086w9s INFO: SuperFetch activated77496 TID-oum086w9s WARN: SuperFetch: recovered 1 jobs77496 TID-oum08eu3o SleepWorker JID-f002a41393f9a79a4366d2b5 INFO: startStarted CFinished C77496 TID-oum08eu3o SleepWorker JID-f002a41393f9a79a4366d2b5 INFO: done: 30.011 sec

Whoa, isn't that cool?

See that line where it says SuperFetch: recovered 1 jobs.

Although the job wasn't pushed back to the queue in Redis,Sidekiq somehow recovered our lost job having ID f002a41393f9a79a4366d2b5and reprocessed that job again!

Interested to learn about how Sidekiq did that? Keep on reading.

Note that, since Sidekiq Pro is a close sourced and commercial software,we cannot explain super_fetch's exact implementation details.

As we discussed in-depth before,Sidekiq's basic_fetch strategy uses BRPOP Redis commandto fetch a job from the queue in Redis.It works great to some extent,but it is prone to losing jobif Sidekiq crashes or does not shutdown gracefully.

On the other hand, Sidekiq Pro offers super_fetch strategy which usesRPOPLPUSH Redis command to fetch a job.

RPOPLPUSH Redis command providesa unique approach towards implementing a reliable queue.RPOPLPUSH command accepts two listsnamely a source list and a destination list.This command atomicallyreturns and removes the last element from the source list,and pushes that element as the first element in the destination list.Atomically means that both pop and push operationsare performed as a single operation at the same time;i.e. both should succeed, otherwise both are treated as failed.

super_fetch registers a private queue in Redisfor each Sidekiq process on start-up.super_fetch atomically fetches a scheduled jobfrom the public queue in Redisand pushes that job into the private queue (or working queue)using RPOPLPUSH Redis command.Once the job is finished processing,Sidekiq removes that job from the private queue.During a graceful shutdown,Sidekiq moves back the unfinished jobsfrom the private queue to the public queue.If shutdown of Sidekiq process is not graceful,the unfinished jobs of that Sidekiq processremain there in the private queue which are called as orphaned jobs.On restarting or starting another Sidekiq process,super_fetch looks for such orphaned jobs in the private queues.If Sidekiq finds orphaned jobs, Sidekiq re-enqueue them and processes again.

It may happen thatwe have multiple Sidekiq processes running at the same time.If a process dies among them, its unfinished jobs become orphans.This Sidekiq wikidescribes in detail the criteria which super_fetch relies uponfor identifying which jobs are orphaned and which jobs are not orphaned.If we don't restart or start another process,super_fetch may take 5 minutes or 3 hours to recover such orphaned jobs.The recommended approach is to restart or start another Sidekiq processto signal super_fetch to look for orphans.

Interestingly, in the older versions of Sidekiq Pro,super_fetch performed checks for orphaned jobs and queuesevery 24 hoursat the Sidekiq process startup.Due to this, when the Sidekiq process crashes,the orphaned jobs of that process remain unpicked for up to 24 hoursuntil the next restart.This orphan delay check windowhad been later lowered to 1 hour in Sidekiq Pro 3.4.1.

Another fun thing to know is that,there existed two fetch strategies namelyreliable_fetchand timed_fetchin the older versions of Sidekiq Pro.Apparently, reliable_fetchdid not work with Dockerand timed_fetch had asymptotic computational complexity O(log N),comparativelyless efficientthan super_fetch,which has asymptotic computational complexity O(1).Both of these strategies had been deprecatedin Sidekiq Pro 3.4.0 in favor of super_fetch.Later, both of these strategies had beenremovedin Sidekiq Pro 4.0and are not documented anywhere.

Final result

We have enabled super_fetch in our application andit seemed to be working without any major issues so far.Our Kubernetes background pods does not seem tobe loosing any jobs when these pods are restarted.

Update : Mike Pheram, author of Sidekiq, posted followingcomment.

Faktory provides all of the beanstalkd functionality, including the same reliability, with a nicer Web UI. It's free and OSS.https://github.com/contribsys/faktory http://contribsys.com/faktory/

Let's suppose we need to run migrations using rake db:migrate in a Railsproject setup script. We can use the Kernel#system method.

irb> system('rake db:migrate')

Ruby 2.5.0

Executing system returns true or false. Another feature of system isthat it eats up the exceptions.

Let's suppose our migrations can run successfully. In this case the systemcommand for running migrations will return true.

irb> system('rake db:migrate') => true

Let's suppose we have a migration that is trying to add a column to a tablewhich does not exist. In this case, the system command for running migrationswill return false.

irb> system('rake db:migrate')== 20180311211836 AddFirstNameToAdmins: migrating =============================-- add_column(:admins, :first_name, :string)rake aborted!StandardError: An error has occurred, this and all later migrations canceled:PG::UndefinedTable: ERROR:  relation "admins" does not exist: ALTER TABLE "admins" ADD "first_name" character varying...Tasks: TOP => db:migrate(See full trace by running task with --trace) => false

As we can see, even when there is a failure in executing system commands, thereturn value is false. Ruby does not raise an exception in those cases.

However, we can use raise explicitly to raise an exception and halt the setupscript execution.

irb> system('rake db:migrate') || raise('Failed to run migrations')== 20180311211836 AddFirstNameToAdmins: migrating =============================-- add_column(:admins, :first_name, :string)rake aborted!StandardError: An error has occurred, this and all later migrations canceled:PG::UndefinedTable: ERROR:  relation "admins" does not exist: ALTER TABLE "admins" ADD "first_name" character varying...Tasks: TOP => db:migrate(See full trace by running task with --trace)Traceback (most recent call last):        2: from /Users/amit/.rvm/rubies/ruby-2.5.0/bin/irb:11:in `<main>'        1: from (irb):4RuntimeError (Failed to run migrations)

Ruby 2.6.0-preview1

Ruby 2.6 make our lives easier by providing an option exception: true so thatwe do not need to use raise explicitly to halt script execution.

irb> system('rake db:migrate', exception: true)== 20180311211836 AddFirstNameToAdmins: migrating =============================-- add_column(:admins, :first_name, :string)rake aborted!StandardError: An error has occurred, this and all later migrations canceled:PG::UndefinedTable: ERROR:  relation "admins" does not exist: ALTER TABLE "admins" ADD "first_name" character varying...Tasks: TOP => db:migrate(See full trace by running task with --trace)Traceback (most recent call last):        3: from /Users/amit/.rvm/rubies/ruby-2.6.0-preview1/bin/irb:11:in `<main>'        2: from (irb):2        1: from (irb):2:in `system'RuntimeError (Command failed with exit 1: rake)

Ruby 2.6 works the same way as previous Ruby versions when used without theexception option or used with exception set as false.

irb> system('rake db:migrate', exception: false)== 20180311211836 AddFirstNameToAdmins: migrating =============================-- add_column(:admins, :first_name, :string)rake aborted!StandardError: An error has occurred, this and all later migrations canceled:PG::UndefinedTable: ERROR:  relation "admins" does not exist: ALTER TABLE "admins" ADD "first_name" character varying...Tasks: TOP => db:migrate(See full trace by running task with --trace) => false

Here is the relevant commitand discussion for this change.

system is not the only way to execute scripts like these. We wrotea blog6 years ago which discusses the differences between running commands usingbacktick, exec, sh, popen3, popen2e and Process.spawn.

The Chinese version of this blog is availablehere.

division_thread = Thread.new do  puts "Calculating 4/0 in division_thread"  puts "Result is: #{4/0}"  puts "Exiting from division_thread"endsleep 1puts "In the main thread"

Execution of it looks like this.

$ RBENV_VERSION=2.4.0 ruby thread_example_1.rbCalculating 4/0 in division_threadIn the main thread

Note that the last two lines from the block were not printed. Also notice thatafter failing in the thread the program continued to run in main thread. That'swhy we got the message "In the main thread".

This is because the default behavior of Ruby is to silently ignore exceptions inthreads and then to continue to execute in the main thread.

Enabling abort_on_exception to stop on failure

If we want an exception in a thread to stop further processing both in thethread and in the main thread then we can enable Thread[.#]abort_on_exceptionon that thread to achieve that.

Notice that in the below code we are using Thread.current.

division_thread = Thread.new do  Thread.current.abort_on_exception = true  puts "Calculating 4/0 in division_thread"  puts "Result is: #{4/0}"  puts "Exiting from division_thread"endsleep 1puts "In the main thread"

$ RBENV_VERSION=2.4.0 ruby thread_example_2.rbCalculating 4/0 in division_threadthread_example_2.rb:5:in `/': divided by 0 (ZeroDivisionError)  from thread_example_2.rb:5:in `block in <main>'

As we can see once an exception was encountered in the thread then processingstopped on both in the thread and in the main thread.

Note that Thread.current.abort_on_exception = true activates this behavioronly for the current thread.

If we want this behavior globally for all the threads then we need to useThread.abort_on_exception = true.

Running program with debug flag to stop on failure

Let's run the original code with --debug option.

$ RBENV_VERSION=2.4.0 ruby --debug thread_example_1.rbthread_example_1.rb:1: warning: assigned but unused variable - division_threadCalculating 4/0 in division_threadException `ZeroDivisionError' at thread_example_1.rb:3 - divided by 0Exception `ZeroDivisionError' at thread_example_1.rb:7 - divided by 0thread_example_1.rb:3:in `/': divided by 0 (ZeroDivisionError)  from thread_example_1.rb:3:in `block in <main>'

In this case the exception is printed in detail and the code in main thread wasnot executed.

Usually when we execute a program with --debug option then the behavior of theprogram does not change. We expect the program to print more stuff but we do notexpect behavior to change. However in this case the --debug option changes thebehavior of the program.

Running program with join on thread to stop on failure

If a thread raises an exception and abort_on_exception and $DEBUG flags arenot set then that exception will be processed at the time of joining of thethread.

division_thread = Thread.new do  puts "Calculating 4/0 in division_thread"  puts "Result is: #{4/0}"  puts "Exiting from division_thread"enddivision_thread.joinputs "In the main thread"

$ RBENV_VERSION=2.4.0 ruby thread_example_3.rbCalculating 4/0 in division_threadthread_example_3.rb:3:in `/': divided by 0 (ZeroDivisionError)  from thread_example_3.rb:3:in `block in <main>'

Both Thread#join and Thread#value will stop processing in the thread and inthe main thread once an exception is encountered.

Introduction of report_on_exception in Ruby 2.4

Almost 6 years ago, Charles Nutter (headius) hadproposed that the exceptions raised in threads should be automatically loggedand reported, by default. To make his point, he explained issues similar to whatwe discussed above about the Ruby's behavior of silently ignoring exceptions inthreads. Here is the relevantdiscussion on his proposal.

Following are some of the notable points discussed.

• Enabling Thread[.#]abort_on_exception, by default, is not always a goodidea.
• There should be a flag which, if enabled, would print the thread-killingexception info.
• In many cases, people spawn one-off threads which are not hard-referencedusing Thread#join or Thread#value. Such threads gets garbage collected.Should it report the thread-killing exception at the time of garbagecollection if such a flag is enabled?
• Should it warn usingWarning#warnor redirect to STDERR device while reporting?

Charles Nutter suggested that a configurable global flagThread.report_on_exception and instance-level flagThread#report_on_exception should be implemented having its default value astrue. When set to true, it should report print exception information.

Matz and other core members approved that Thread[.#]report_on_exception can beimplemented having its default value set to false.

Charles Nutter, Benoit Daloze and other people demanded that it should be trueby default so that programmers can be aware of the silently disappearing threadsbecause of exceptions.

Shyouhei Urabe advised thatdue to some technical challenges, the default value should be set to false soas this feature could land in Ruby. Once this feature is in then the defaultvalue can be changed in a later release.

Nobuyoshi Nakada (nobu) pushed animplementationfor Thread[.#]report_on_exception with a default value set to false. It wasreleased in Ruby 2.4.0.

Let's try enabling report_on_exception globally usingThread.report_on_exception.

Thread.report_on_exception = truedivision_thread = Thread.new do  puts "Calculating 4/0 in division_thread"  puts "Result is: #{4/0}"  puts "Exiting from division_thread"endaddition_thread = Thread.new do  puts "Calculating nil+4 in addition_thread"  puts "Result is: #{nil+4}"  puts "Exiting from addition_thread"endsleep 1puts "In the main thread"

$ RBENV_VERSION=2.4.0 ruby thread_example_4.rbCalculating 4/0 in division_thread#<Thread:0x007fb10f018200@thread_example_4.rb:3 run> terminated with exception:thread_example_4.rb:5:in `/': divided by 0 (ZeroDivisionError)  from thread_example_4.rb:5:in `block in <main>'Calculating nil+4 in addition_thread#<Thread:0x007fb10f01aca8@thread_example_4.rb:9 run> terminated with exception:thread_example_4.rb:11:in `block in <main>': undefined method `+' for nil:NilClass (NoMethodError)In the main thread

It now reports the exceptions in all threads. It prints that theThread:0x007fb10f018200 wasterminated with exception: divided by 0 (ZeroDivisionError). Similarly,another thread Thread:0x007fb10f01aca8 wasterminated with exception: undefined method '+' for nil:NilClass (NoMethodError).

Instead of enabling it globally for all threads, we can enable it for aparticular thread using instance-level Thread#report_on_exception.

division_thread = Thread.new do  puts "Calculating 4/0 in division_thread"  puts "Result is: #{4/0}"  puts "Exiting from division_thread"endaddition_thread = Thread.new do  Thread.current.report_on_exception = true  puts "Calculating nil+4 in addition_thread"  puts "Result is: #{nil+4}"  puts "Exiting from addition_thread"endsleep 1puts "In the main thread"

In the above case we have enabled report_on_exception flag just foraddition_thread.

Let's execute it.

$ RBENV_VERSION=2.4.0 ruby thread_example_5.rbCalculating 4/0 in division_threadCalculating nil+4 in addition_thread#<Thread:0x007f8e6b007f70@thread_example_5.rb:7 run> terminated with exception:thread_example_5.rb:11:in `block in <main>': undefined method `+' for nil:NilClass (NoMethodError)In the main thread

Notice how it didn't report the exception which killed thread division_thread.As expected, it reported the exception that killed thread addition_thread.

With the above changes ruby reports the exception as soon as it encounters.However if these threads are joined then they will still raise exception.

division_thread = Thread.new do  Thread.current.report_on_exception = true  puts "Calculating 4/0 in division_thread"  puts "Result is: #{4/0}"  puts "Exiting from division_thread"endbegin  division_thread.joinrescue => exception  puts "Explicitly caught - #{exception.class}: #{exception.message}"endputs "In the main thread"

$ RBENV_VERSION=2.4.0 ruby thread_example_6.rbCalculating 4/0 in division_thread#<Thread:0x007f969d00d828@thread_example_6.rb:1 run> terminated with exception:thread_example_6.rb:5:in `/': divided by 0 (ZeroDivisionError)  from thread_example_6.rb:5:in `block in <main>'Explicitly caught - ZeroDivisionError: divided by 0In the main thread

See how we were still be able to handle the exception raised indivision_thread above after joining it despite it reported it before due toThread#report_on_exception flag.

report_on_exception defaults to true in Ruby 2.5

Benoit Daloze (eregon) strongly advocated that boththe Thread.report_on_exception and Thread#report_on_exception should havedefault value as true. Here is therelevant feature request.

After approval from Matz,Benoit Daloze pushed theimplementationby fixing the failing tests and silencing the unnecessary verbose warnings.

It was released as part of Ruby 2.5.

Now in ruby 2.5 we can simply write like this.

division_thread = Thread.new do  puts "Calculating 4/0 in division_thread"  puts "Result is: #{4/0}"  puts "Exiting from division_thread"endaddition_thread = Thread.new do  puts "Calculating nil+4 in addition_thread"  puts "Result is: #{nil+4}"  puts "Exiting from addition_thread"endsleep 1puts "In the main thread"

Let's execute it with Ruby 2.5.

$ RBENV_VERSION=2.5.0 ruby thread_example_7.rbCalculating 4/0 in division_thread#<Thread:0x00007f827689a238@thread_example_7.rb:1 run> terminated with exception (report_on_exception is true):Traceback (most recent call last):  1: from thread_example_7.rb:3:in `block in <main>'thread_example_7.rb:3:in `/': divided by 0 (ZeroDivisionError)Calculating nil+4 in addition_thread#<Thread:0x00007f8276899b58@thread_example_7.rb:7 run> terminated with exception (report_on_exception is true):Traceback (most recent call last):thread_example_7.rb:9:in `block in <main>': undefined method `+' for nil:NilClass (NoMethodError)In the main thread

We can disable the thread exception reporting globally usingThread.report_on_exception = false or for a particular thread usingThread.current.report_on_exception = false.

Future Possibilities

In addition to this feature, Charles Nutter alsosuggested that it will be goodif there exists a callback handler which can accept a block to be executed whena thread dies due to an exception. The callback handler can be at global levelor it can be for a specific thread.

Thread.on_exception do  # some stuffend

In the absence of such handler libraries need to resort to custom code to handleexceptions.Here is howSidekiq handles exceptions raised in threads.

Important thing to note is that report_on_exception does not change behaviorof the code. It does more reporting when a thread dies and when it comes tothread dies more reporting is a good thing.

Before Ruby 2.5

Before Ruby 2.5, we could measure just the line coverage using Coverage.

Line coverage tells us whether a line is executed or not. If executed, then howmany times that line was executed.

We have a file called score.rb.

score = 33if score >= 40  p :PASSEDelse  p :FAILEDend

Now create another file score_coverage.rb.

require "coverage"Coverage.startload "score.rb"p Coverage.result

We used Coverage#start method to measure the coverage of score.rb file.Coverage#result returns the coverage result.

Let's run it with Ruby 2.4.

$ RBENV_VERSION=2.4.0 ruby score_coverage.rb:FAILED{ "score.rb"=> [1, nil, 1, 0, nil, 1, nil] }

Let's look at the output. Each value in the array [1, nil, 1, 0, nil, 1, nil]denotes the count of line executions by the interpreter for each line inscore.rb file.

This array is also called the "line coverage" of score.rb file.

A nil value in line coverage array means coverage is disabled for thatparticular line number or it is not a relevant line. Lines like else, endand blank lines have line coverage disabled.

Here's how we can read above line coverage result.

• Line number 1 (i.e. 0th index in the above result array) was executed once.
• Coverage was disabled for line number 2 (i.e. index 1) as it is blank.
• Line number 3 (i.e. index 2) was executed once.
• Line number 4 did not execute.
• Coverage was disabled for line number 5 as it contains only else clause.
• Line number 6 was executed once.
• Coverage was disabled for line number 7 as it contains just end keyword.

After Ruby 2.5

There was a pull request opened in 2014to add method coverage and decision coverage metrics in Ruby. It wasrejected byYusuke Endoh as he saw some issues with it andmentioned that he was also working on a similar implementation.

In Ruby 2.5, Yusuke Endohadded branch coverage and method coverage featureto the Coverage library.

Let's see what's changed in Coverage library in Ruby 2.5.

Line Coverage

If we execute above example using Ruby 2.5, we will see no change in the result.

$ RBENV_VERSION=2.5.0 ruby score_coverage.rb:FAILED{ "score.rb" => [1, nil, 1, 0, nil, 1, nil] }

This behavior is maintained to ensure that the Coverage#start API stays 100%backward compatible.

If we explicitly enable lines option on Coverage#start method in the abovescore_coverage.rb file, the coverage result will be different now.

require "coverage"Coverage.start(lines: true)load "score.rb"p Coverage.result

$ RBENV_VERSION=2.5.0 ruby score_coverage.rb:FAILED{ "score.rb" => {    :lines => [1, nil, 1, 0, nil, 1, nil]  }}

We can see that the coverage result is now a hash which reads that thescore.rb file has lines coverage as [1, nil, 1, 0, nil, 1, nil].

Branch Coverage

Branch coverage helps us identify which branches are executed and which ones arenot executed.

Let's see how to get branch coverage.

We will update the score_coverage.rb by enabling branches option.

require "coverage"Coverage.start(branches: true)load "score.rb"p Coverage.result

$ RBENV_VERSION=2.5.0 ruby score_coverage.rb:FAILED{ "score.rb" =>  { :branches => {      [:if, 0, 3, 0, 7, 3] => {        [:then, 1, 4, 2, 4, 15] => 0,        [:else, 2, 6, 2, 6, 15] => 1      }    }  }}

Here is how to read the data in array.

[  BRANCH_TYPE,  UNIQUE_ID,  START_LINE_NUMBER,  START_COLUMN_NUMBER,  END_LINE_NUMBER,  END_COLUMN_NUMBER]

Please note that column numbers start from 0 and line numbers start from 1.

Let's try to read above printed branch coverage result.

[:if, 0, 3, 0, 7, 3] reads that if statement starts at line 3 & column 0 andends at line 7 & column 3.

[:then, 1, 4, 2, 4, 15] reads that then clause starts at line 4 & column 2and ends at line 4 & column 15.

Similarly, [:else, 2, 6, 2, 6, 15] reads that else clause starts at line 6 &column 2 and ends at line 6 & column 15.

Most importantly as per the branch coverage format, we can see that the branchfrom if to then was never executed since COUNTER is 0. The anotherbranch from if to else was executed once since COUNTER is 1.

Method Coverage

Measuring method coverage helps us identify which methods were invoked and whichwere not.

We have a file grade_calculator.rb.

students_scores = { "Sam" => [53, 91, 72],                    "Anna" => [91, 97, 95],                    "Bob" => [33, 69, 63] }def average(scores)  scores.reduce(&:+)/scores.sizeenddef grade(average_score)  case average_score  when 90.0..100.0 then :A  when 80.0..90.0 then :B  when 70.0..80.0 then :C  when 60.0..70.0 then :D  else :F  endenddef greet  puts "Congratulations!"enddef warn  puts "Try hard next time!"endstudents_scores.each do |student_name, scores|  achieved_grade = grade(average(scores))  puts "#{student_name}, you've got '#{achieved_grade}' grade."  if achieved_grade == :A    greet  elsif achieved_grade == :F    warn  end  putsend

To measure method coverage of above file, let's creategrade_calculator_coverage.rb by enabling methods option on Coverage#startmethod.

require "coverage"Coverage.start(methods: true)load "grade_calculator.rb"p Coverage.result

Let's run it using Ruby 2.5.

$ RBENV_VERSION=2.5.0 ruby grade_calculator_coverage.rbSam, you've got 'C' grade.Anna, you've got 'A' grade.Congratulations!Bob, you've got 'F' grade.Try hard next time!{ "grade_calculator.rb" => {    :methods => {      [Object, :warn, 23, 0, 25, 3] => 1,      [Object, :greet, 19, 0, 21, 3] => 1,      [Object, :grade, 9, 0, 17, 3] => 3,      [Object, :average, 5, 0, 7, 3] => 3    }  }}

The format of method coverage result is defined as shown below.

[ CLASS_NAME,  METHOD_NAME,  START_LINE_NUMBER,  START_COLUMN_NUMBER,  END_LINE_NUMBER,  END_COLUMN_NUMBER ]

Therefore, [Object, :grade, 9, 0, 17, 3] => 3 reads that the Object#grademethod which starts from line 9 & column 0 to line 17 & column 3 was invoked 3times.

Conclusion

We can measure all coverages at once also.

Coverage.start(lines: true, branches: true, methods: true)

What's the use of these different types of coverages anyway?

Well, one use case is to integrate this in a test suite and to determine whichlines, branches and methods are executed and which ones are not executed by thetest. Further, we can sum up these and evaluate total coverage of a test suite.

Author of this feature, Yusuke Endoh has releasedcoverage-helpers gem which allowsfurther advanced manipulation and processing of coverage results obtained usingCoverage#result.

class AverageService  attr_reader :numbers, :coerced_numbers  def initialize(numbers)    @numbers = numbers    @coerced_numbers = coerce_numbers  end  def average    sum / count  end  private  def coerce_numbers    numbers.map do |number|      begin        Float(number)      rescue Exception => exception        puts "#{exception.message} (#{exception.class})\n\t#{exception.backtrace.join("\n\t")}"        puts "Coercing '#{number}' as 0.0\n\n"        0.0      end    end  end  def sum    coerced_numbers.map(&:to_f).sum  end  def count    coerced_numbers.size.to_f  endendaverage = AverageService.new(ARGV).averageputs "Average is: #{average}"

$ RBENV_VERSION=2.4.0 ruby average_service.rb 5 4f 7 1s0invalid value for Float(): "4f" (ArgumentError)average_service.rb:18:in `Float'average_service.rb:18:in `block in coerce_numbers'average_service.rb:16:in `map'average_service.rb:16:in `coerce_numbers'average_service.rb:6:in `initialize'average_service.rb:37:in `new'average_service.rb:37:in `<main>'Coercing '4f' as 0.0invalid value for Float(): "1s0" (ArgumentError)average_service.rb:18:in `Float'average_service.rb:18:in `block in coerce_numbers'average_service.rb:16:in `map'average_service.rb:16:in `coerce_numbers'average_service.rb:6:in `initialize'average_service.rb:37:in `new'average_service.rb:37:in `<main>'Coercing '1s0' as 0.0Average of [5.0, 0.0, 7.0, 0.0] is: 3.0

It was proposed that there should bea simple method to print the caught exception using the same format that rubyuses while printing an uncaught exception.

Some of the proposed method names were display, formatted, to_formatted_s,long_message, and full_message.

Matz approved theException#full_message method name.

In Ruby 2.5, we can re-write above example as follows.

class AverageService  attr_reader :numbers, :coerced_numbers  def initialize(numbers)    @numbers = numbers    @coerced_numbers = coerce_numbers  end  def average    sum / count  end  private  def coerce_numbers    numbers.map do |number|      begin        Float(number)      rescue Exception => exception        puts exception.full_message        puts "Coercing '#{number}' as 0.0\n\n"        0.0      end    end  end  def sum    coerced_numbers.map(&:to_f).sum  end  def count    coerced_numbers.size.to_f  endendaverage = AverageService.new(ARGV).averageputs "Average is: #{average}"

$ RBENV_VERSION=2.5.0 ruby average_service.rb 5 4f 7 1s0Traceback (most recent call last):6: from average_service.rb:37:in `<main>'5: from average_service.rb:37:in `new'4: from average_service.rb:6:in `initialize'3: from average_service.rb:16:in `coerce_numbers'2: from average_service.rb:16:in `map'1: from average_service.rb:18:in `block in coerce_numbers'average_service.rb:18:in `Float': invalid value for Float(): "4f" (ArgumentError)Coercing '4f' as 0.0Traceback (most recent call last):6: from average_service.rb:37:in `<main>'5: from average_service.rb:37:in `new'4: from average_service.rb:6:in `initialize'3: from average_service.rb:16:in `coerce_numbers'2: from average_service.rb:16:in `map'1: from average_service.rb:18:in `block in coerce_numbers'average_service.rb:18:in `Float': invalid value for Float(): "1s0" (ArgumentError)Coercing '1s0' as 0.0Average of [5.0, 0.0, 7.0, 0.0] is: 3.0

Note that, Ruby 2.5 prints exception backtrace in reverse order if STDERR isunchanged and is a TTY as discussedin our previous blog post.

Before Ruby 2.5

Before Ruby 2.5, the printed backtrace contained the exception class and theerror message at the top. Next line contained where in the program the exceptionwas raised. Next we got more lines which contained cascaded method calls.

Consider a simple Ruby program.

class DivisionService  attr_reader :a, :b  def initialize(a, b)    @a, @b = a.to_i, b.to_i  end  def divide    puts a / b  endendDivisionService.new(ARGV[0], ARGV[1]).divide

Let's execute it using Ruby 2.4.

$ RBENV_VERSION=2.4.0 ruby division_service.rb 5 0division_service.rb:9:in `/': divided by 0 (ZeroDivisionError)from division_service.rb:9:in `divide'from division_service.rb:13:in `<main>'

In the printed backtrace above, the first line shows the location, error messageand the exception class name; whereas the subsequent lines shows the callermethod names and their locations. Each line in the backtrace above is oftenconsidered as a stack frame placed on the call stack.

Most of the times, a backtrace has so many lines that it makes it very difficultto fit the whole backtrace in the visible viewport of the terminal.

Since the backtrace is printed in top to bottom order the meaningful informationlike error message, exception class and the exact location where the exceptionwas raised is displayed at top of the backtrace. It means developers often needto scroll to the top in the terminal window to find out what went wrong.

After Ruby 2.5

Over 4 years ago an issue was createdto make printing of backtrace in reverse order configurable.

After much discussion Nobuyoshi Nakada made the commit to print backtrace anderror messagein reverse orderonly when the error output device (STDERR) is a TTY (i.e. a terminal). Messagewill not be printed in reverse order if the original STDERR is attached tosomething like a File object.

Look at the code herewhere the check happens if STDERR is a TTY and is unchanged.

Let's execute the same program using Ruby 2.5.

$ RBENV_VERSION=2.5.0 ruby division_service.rb 5 0Traceback (most recent call last):2: from division_service.rb:13:in `<main>'1: from division_service.rb:9:in `divide'division_service.rb:9:in `/': divided by 0 (ZeroDivisionError)$

We can notice two new changes in the above backtrace.

1. The error message and exception class is printed last (i.e. at the bottom).
2. The stack alsoadds frame numberwhen printing in reverse order.

This feature makes the debugging convenient when the backtrace size is a quitebig and cannot fit in the terminal window. We can easily see the error messagewithout scrolling up now.

Note that, the Exception#backtrace attribute still holds an array of stackframes like before in the top to bottom order.

So if we rescue the caught exception and print the backtrace manually

class DivisionService  attr_reader :a, :b  def initialize(a, b)    @a, @b = a.to_i, b.to_i  end  def divide    puts a / b  endendbegin  DivisionService.new(ARGV[0], ARGV[1]).dividerescue Exception => e  puts "#{e.class}: #{e.message}"  puts e.backtrace.join("\n")end

we will get the old behavior.

$ RBENV_VERSION=2.5.0 ruby division_service.rb 5 0ZeroDivisionError: divided by 0division_service.rb:9:in `/'division_service.rb:9:in `divide'division_service.rb:16:in `<main>'$

Also, note that if we assign STDERR with a File object, thus making it anon-TTY

puts "STDERR is a TTY? [before]: #{$stderr.tty?}"$stderr = File.new("stderr.log", "w")$stderr.sync = trueputs "STDERR is a TTY? [after]: #{$stderr.tty?}"class DivisionService  attr_reader :a, :b  def initialize(a, b)    @a, @b = a.to_i, b.to_i  end  def divide    puts a / b  endendDivisionService.new(ARGV[0], ARGV[1]).divide

we can get the old behavior but the backtrace would be written to the specifiedfile and not to STDERR.

$ RBENV_VERSION=2.5.0 ruby division_service.rb 5 0STDERR is a TTY? [before]: trueSTDERR is a TTY? [after]: false$ cat stderr.logdivision_service.rb:14:in `/': divided by 0 (ZeroDivisionError)from division_service.rb:14:in `divide'from division_service.rb:18:in `<main>'$

This feature has been tagged asexperimental feature.What it means is that Ruby team isgathering feedbackon this feature.

Let's say we have a hash{ id: 1, name: 'Ruby 2.5', description: 'BigBinary Blog' } and we want toselect key value pairs having the keys name and description.

We can use theHash#select method.

irb> blog = { id: 1, name: 'Ruby 2.5', description: 'BigBinary Blog' }  => {:id=>1, :name=>"Ruby 2.5", :description=>"BigBinary Blog"}irb> blog.select { |key, value| [:name, :description].include?(key) }  => {:name=>"Ruby 2.5", :description=>"BigBinary Blog"}

Matzbara Masanao proposed a simplemethod to take care of this problem.

Some of the names proposed were choice and pick.

Matz suggested the name slice since thismethod is ActiveSupport compatible.

Ruby 2.5.0

irb> blog = { id: 1, name: 'Ruby 2.5', description: 'BigBinary Blog' }  => {:id=>1, :name=>"Ruby 2.5", :description=>"BigBinary Blog"}irb> blog.slice(:name, :description)  => {:name=>"Ruby 2.5", :description=>"BigBinary Blog"}

So, now we can use a simple method slice to select key value pairs from a hashwith specified keys.

Here is the relevant commitand discussion.

Customer = Struct.new(:name, :email)Customer.new("John", "john@example.com")

This approach works when the arguments list is short. When arguments listincreases then it gets harder to track which position maps to which value.

Here if we pass keyword argument then we won't get any error. But the values arenot what we wanted.

Customer.new(name: "John", email: "john@example.com")=> #<struct Customer name={:name=>"John", :email=>"john@example.com"}, email=nil>

Ruby 2.5 introducedcreating structs using keyword arguments.Relevant pull request is here.

However this introduces a problem. How do we indicate to Struct if we want topass arguments using position or keywords.

Takashi Kokubunsuggestedto use keyword_argument as an identifier.

Customer = Struct.new(:name, :email, keyword_argument: true)Customer.create(name: "John", email: "john@example.com")

Matzsuggestedto change the name to keyword_init.

So in Ruby 2.5 we can create structs using keywords as long as we are passingkeyword_init.

Customer = Struct.new(:name, :email, keyword_init: true)Customer.new(name: "John", email: "john@example.com")=> #<struct Customer name="John", email="john@example.com">

In Ruby 2.5, a similar methodHash#transform_keys is added fortransforming keys of the hash.

>> h = { name: "John", email: "john@example.com" }=> {:name=>"John", :email=>"john@example.com"}>> h.transform_keys { |k| k.to_s }=> {"name"=>"John", "email"=>"john@example.com"}

The bang sibling of this method, Hash#transform_keys! is also added whichchanges the hash in place.

These two methods are already present inActive Support from Railsand are natively supported in Ruby now.

Rails master is already supportingusing the native methods ifsupported by the Ruby version.

Ruby has sequence predicates such as all?, none?, one? and any? whichtake a block and evaluate that by passing every element of the sequence to it.

if queries.any? { |sql| /LEFT OUTER JOIN/i =~ sql }logger.log "Left outer join detected"end

Ruby 2.5 allows using a shorthand for this bypassing a pattern argument.Internally case equality operator(===) is used against every element of thesequence and the pattern argument.

if queries.any?(/LEFT OUTER JOIN/i)logger.log "Left outer join detected"end# Translates to:queries.any? { |sql| /LEFT OUTER JOIN/i === sql }

This allows us to write concise and shorthand expressions where block is onlyused for comparisons. This feature is applicable to all?, none?, one? andany? methods.

Similarities with Enumerable#grep

This feature is based on how Enumerable#grep works. grep returns an array ofevery element in the sequence for which the case equality operator(===)returns true by applying the pattern. In this case, the all? and friendsreturn true or false.

There is a proposal to add it forselect and reject as well.

Ruby allows pretty printing of objects usingpp method.

Before Ruby 2.5, we had to require PP explicitly before using it. Even theofficial documentation states that "All examples assume you have loaded the PPclass with require 'pp'".

>> months = %w(January February March)=> ["January", "February", "March"]>> pp monthsNoMethodError: undefined method `pp' for main:ObjectDid you mean?  pfrom (irb):5from /Users/prathamesh/.rbenv/versions/2.4.1/bin/irb:11:>> require 'pp'=> true>> pp months["January", "February", "March"]=> ["January", "February", "March"]

In Ruby 2.5, we don't need to require pp. Itgets required by default. We can useit directly.

>> months = %w(January February March)=> ["January", "February", "March"]>> pp months["January", "February", "March"]=> ["January", "February", "March"]

This feature was added after Ruby 2.5.0 preview 1 was released, so it's notpresent in the preview. It's present inRuby trunk.

The names of these methods are not very intuitive. Active Support from Railsalready has aliases forthe unshift and push methods, namely prepend and append.

In Ruby 2.5, these methodsare added in the Ruby language itself.

>> a = ["hello"]=> ["hello"]>> a.append "world"=> ["hello", "world"]>> a.prepend "Hey"=> ["Hey", "hello", "world"]>>

They are implemented as aliases to the original unshift and push methods sothere is no change in the behavior.

irb> "Hello".yield_self { |str| str + " World" }  => "Hello World"

How is it different from try in Rails ?

Without a method argument trybehaves similar to yield_self. It would yield to the given block unless thereceiver is nil and returns the output of the last statement in the block.

irb> "Hello".try { |str| str + " World" }  => "Hello World"

Couple of differences to note are, try is not part of Ruby but Rails. Alsotry's main purpose is protection against nil hence it doesn't execute theblock if receiver is nil.

irb> nil.yield_self { |obj| "Hello World" }  => "Hello World"irb> nil.try { |obj| "Hello World" }  => nil

What about tap?

tap also is similar to yield_self. It's part of Ruby itself. The onlydifference is the value that is returned. tap returns the receiver itselfwhile yield_self returns the output of the block.

irb> "Hello".yield_self { |str| str + " World" }  => "Hello World"irb> "Hello".tap { |str| str + " World" }  => "Hello"

Overall, yield_self improves readability of the code by promoting chainingover nested function calls. Here is an example of both the styles.

irb> add_greeting = -> (str) { "HELLO " + str }irb> to_upper = -> (str) { str.upcase }# with new `yield_self`irb> "world".yield_self(&to_upper)            .yield_self(&add_greeting)  => "HELLO WORLD"# nested function callsirb> add_greeting.call(to_upper.call("world"))  => "HELLO WORLD"

yield_self is part of Kernel and hence it's available to all the objects.

Let's say that we have a string Projects::CategoriesController and we want toremove Controller. We can usechomp method.

irb> "Projects::CategoriesController".chomp("Controller")=> "Projects::Categories"

However if we want to remove Projects:: from the string then there is nocorresponding method of chomp. We need to resort tosub.

irb> "Projects::CategoriesController".sub(/Projects::/, '')=> "CategoriesController"

Naotoshi Seo did not like using regularexpression for such a simple task. He proposed that Ruby should have a methodfor taking care of such tasks.

Some of the names proposed were remove_prefix, deprefix, lchomp,remove_prefix and head_chomp.

Matz suggested the name delete_prefix andthis method was born.

Ruby 2.5.0-preview1

irb> "Projects::CategoriesController".delete_prefix("Projects::")=> "CategoriesController"

Now in order to delete prefix we can use delete_prefix and to delete suffix wecould use chomp. This did not feel right. So for symmetry delete_suffix wasadded.

irb> "Projects::CategoriesController".delete_suffix("Controller")=> "Projects::Categories"

Read up on this discussion to learnmore about how elixir, go, python, and PHP deal with similar requirements.

> Dir.entries("/Users/john/Desktop/test")> => [".", "..", ".config", "program.rb", "group.txt"]>

We also have methodDir.foreach whichiterates and yields each value from the output of ls -a command to the block.

> Dir.foreach("/Users/john/Desktop/test") { |child| puts child }> .> ..> .config> program.rb> group.txt> test2>

We can see that the output includes the directives for current directory andparent directory which are "." and "..".

When we want to have access only to the children files and directories, we donot need the [".", ".."] subarray.

This is a very common use case and we'll probably have to do something likeDir.entries(path) - [".", ".."] to achieve the desired output.

To overcome such issues,Ruby 2.5 introduced Dir.children. Itreturns the output of ls -a command without the directives for current andparent directories.

> Dir.children("/Users/mohitnatoo/Desktop/test")> => [".config", "program.rb", "group.txt"]>

Additionally, we can use Dir.each_child method to avoid yielding current andparent directory directives while iterating,

> Dir.each_child("/Users/mohitnatoo/Desktop/test") { |child| puts child }> .config> program.rb> group.txt> test2>

As noted in the discussion the nameswere chosen to match with existing methodsPathname#childrenandPathname#each_child.

These additions seem like simple features. Well the issue was posted more thantwo years ago.

irb> array_from_user = [4, 2, 0, 1]  => [4, 2, 0, 1]irb> array_from_user.each do |number|irb>   p 10 / numberirb> rescue ZeroDivisionError => exceptionirb>   p exceptionirb>   nextirb> endSyntaxError: (irb):4: syntax error, unexpected keyword_rescue,expecting keyword_endrescue ZeroDivisionError => exception      ^

Ruby 2.4 throws an error when we try to use rescue/else/ensure inside do/endblocks.

Ruby 2.5.0-preview1

irb> array_from_user = [4, 2, 0, 1]  => [4, 2, 0, 1]irb> array_from_user.each do |number|irb>   p 10 / numberirb> rescue ZeroDivisionError => exceptionirb>   p exceptionirb>   nextirb> end25#<ZeroDivisionError: divided by 0>10 => [4, 2, 0, 1]

Ruby 2.5 supports rescue/else/ensure inside do/end blocks.

Here is relevant commit anddiscussion.

irb> class Projectirb> end=> nilirb> class Categoryirb> end=> nilirb> Project::Category(irb):5: warning: toplevel constant Category referenced by Project::Category => Category

Ruby 2.4 returns the top level constant with a warning if it is unable to find aconstant in the specified scope.

This does not work well in cases where we need constants to be defined with samename at top level and also in the same scope.

Ruby 2.5.0-preview1

irb> class Projectirb> end=> nilirb> class Categoryirb> end=> nilirb> Project::CategoryNameError: uninitialized constant Project::CategoryDid you mean?  Categoryfrom (irb):5

Ruby 2.5 throws an error if it is unable to find a constant in the specifiedscope.

Here is the relevant commitand discussion.

(1..99).min #=> 1(1..99).max #=> 99(1..99).minmax #=> [1, 99]

In Ruby 2.4, Enumurable#min, Enumurable#maxmethods andEnumurable#minmax methodare now more optimized.

We would run the following benchmark snippet for both Ruby 2.3 and Ruby 2.4 andobserve the results

require 'benchmark/ips'Benchmark.ips do |bench|NUM1 = 1_000_000.times.map { rand }ENUM_MIN = Enumerable.instance_method(:min).bind(NUM1)ENUM_MAX = Enumerable.instance_method(:max).bind(NUM1)ENUM_MINMAX = Enumerable.instance_method(:minmax).bind(NUM1)bench.report('Enumerable#min') doENUM_MIN.callendbench.report('Enumerable#max') doENUM_MAX.callendbench.report('Enumerable#minmax') doENUM_MINMAX.callendend

Results for Ruby 2.3

Warming up --------------------------------------Enumerable#min 1.000 i/100msEnumerable#max 1.000 i/100msEnumerable#minmax 1.000 i/100msCalculating -------------------------------------Enumerable#min 14.810 (13.5%) i/s - 73.000 in 5.072666sEnumerable#max 16.131 ( 6.2%) i/s - 81.000 in 5.052324sEnumerable#minmax 11.758 ( 0.0%) i/s - 59.000 in 5.026007s

Ruby 2.4

Warming up --------------------------------------Enumerable#min 1.000 i/100msEnumerable#max 1.000 i/100msEnumerable#minmax 1.000 i/100msCalculating -------------------------------------Enumerable#min 18.091 ( 5.5%) i/s - 91.000 in 5.042064sEnumerable#max 17.539 ( 5.7%) i/s - 88.000 in 5.030514sEnumerable#minmax 13.086 ( 7.6%) i/s - 66.000 in 5.052537s

From the above benchmark results, it can be seen that there has been animprovement in the run times for the methods.

Internally Ruby has changed the logic by which objects are compared, whichresults in these methods being optimized. You can have a look at the commitshere andhere.

Ruby 2.3

CSV::Row.new(%w(banana mango), [1,2]).each #=> #<CSV::Row "banana":1 "mango":2>CSV::Row.new(%w(banana mango), [1,2]).delete_if #=> #<CSV::Row "banana":1 "mango":2>

Some methods raise exception because of this behavior.

> ruby -rcsv -e 'CSV::Table.new([CSV::Row.new(%w{banana mango}, [1, 2])]).by_col.each' #=> /Users/sushant/.rbenv/versions/2.3.0/lib/ruby/2.3.0/csv.rb:850:in `block in each': undefined method `[]' for nil:NilClass (NoMethodError)  from /Users/sushant/.rbenv/versions/2.3.0/lib/ruby/2.3.0/csv.rb:850:in `each'  from /Users/sushant/.rbenv/versions/2.3.0/lib/ruby/2.3.0/csv.rb:850:in `each'  from -e:1:in `<main>'

Ruby 2.4fixed this issue.

Ruby 2.4

CSV::Row.new(%w(banana mango), [1,2]).each #=> #<Enumerator: #<CSV::Row "banana":1 "mango":2>:each>CSV::Row.new(%w(banana mango), [1,2]).delete_if #=> #<Enumerator: #<CSV::Row "banana":1 "mango":2>:delete_if>

As we can see, these methods now return an enumerator when no block is given.

In Ruby 2.4 following code will not raise any exception.

> ruby -rcsv -e 'CSV::Table.new([CSV::Row.new(%w{banana mango}, [1, 2])]).by_col.each'

In Ruby 2.3, these methods convert time into system timezone offset instead ofpreserving timezone offset of the receiver.

Ruby 2.3

> datetime = DateTime.strptime('2017-05-16 10:15:30 +09:00', '%Y-%m-%d %H:%M:%S %Z') #=> #<DateTime: 2017-05-16T10:15:30+09:00 ((2457890j,4530s,0n),+32400s,2299161j)>> datetime.to_time #=> 2017-05-16 06:45:30 +0530> time = Time.new(2017, 5, 16, 10, 15, 30, '+09:00') #=> 2017-05-16 10:15:30 +0900> time.to_time #=> 2017-05-16 06:45:30 +0530

As you can see, DateTime#to_time and Time#to_time methods return time insystem timezone offset +0530.

Ruby 2.4 fixedDateTime#to_timeandTime#to_time.

Now, DateTime#to_time and Time#to_time preserve receiver's timezone offsetinfo.

Ruby 2.4

> datetime = DateTime.strptime('2017-05-16 10:15:30 +09:00', '%Y-%m-%d %H:%M:%S %Z') #=> #<DateTime: 2017-05-16T10:15:30+09:00 ((2457890j,4530s,0n),+32400s,2299161j)>> datetime.to_time #=> 2017-05-16 10:15:30 +0900> time = Time.new(2017, 5, 16, 10, 15, 30, '+09:00') #=> 2017-05-16 10:15:30 +0900> time.to_time #=> 2017-05-16 10:15:30 +0900

Since this is a breaking change for Rails application upgrading to ruby 2.4,Rails 4.2.8 built a compatibility layer by adding aconfig option.ActiveSupport.to_time_preserves_timezone was added to control how to_timehandles timezone offsets.

Here is an example of how application behaves when to_time_preserves_timezoneis set to false.

> ActiveSupport.to_time_preserves_timezone = false> datetime = DateTime.strptime('2017-05-16 10:15:30 +09:00', '%Y-%m-%d %H:%M:%S %Z') #=> Tue, 16 May 2017 10:15:30 +0900> datetime.to_time #=> 2017-05-16 06:45:30 +0530> time = Time.new(2017, 5, 16, 10, 15, 30, '+09:00') #=> 2017-05-16 10:15:30 +0900> time.to_time #=> 2017-05-16 06:45:30 +0530

Here is an example of how application behaves when to_time_preserves_timezoneis set to true.

> ActiveSupport.to_time_preserves_timezone = true> datetime = DateTime.strptime('2017-05-16 10:15:30 +09:00', '%Y-%m-%d %H:%M:%S %Z') #=> Tue, 16 May 2017 10:15:30 +0900> datetime.to_time #=> 2017-05-16 10:15:30 +0900> time = Time.new(2017, 5, 16, 10, 15, 30, '+09:00') #=> 2017-05-16 10:15:30 +0900> time.to_time #=> 2017-05-16 10:15:30 +0900

> 1.dupTypeError: can't dup Fixnumfrom (irb):1:in `dup'from (irb):1

This was confusing because Integer#dup is actually implemented.

> Integer.respond_to? :dup=> true

However, if we were to freeze an Integer it would fail silently.

> 1.freeze=> 1

Ruby 2.4 has now included dup-ability for Integer as well.

> 1.dup=> 1

In Ruby, some object types are immediate variables and therefore cannot beduped/cloned. Yet, there was no graceful way of averting the error thrown by thesanity check when we attempt to dup/clone them.

So now Integer#dup functions exactly the way freeze does -- fail silentlyand return the object itself. It makes sense because nothing about these objectscan be changed in the first place.

In Ruby 2.3, behavior of these methods was inconsistent. Let's see an example.

# Ruby 2.3>> IPAddr.new("1.2.1.3") == "Some ip address"=> IPAddr::InvalidAddressError: invalid address

But if the first argument is invalid IP address and second is valid IP address,then it would return false.

# Ruby 2.3>> "Some ip address" == IPAddr.new("1.2.1.3")=> false

The <=> method would raise exception in both the cases.

# Ruby 2.3>> "Some ip address" <=> IPAddr.new("1.2.1.3")=> IPAddr::InvalidAddressError: invalid address>> IPAddr.new("1.2.1.3") <=> "Some ip address"=> IPAddr::InvalidAddressError: invalid address

In Ruby 2.4, this issue isfixedfor both the methods to return the result without raising exception, if theobjects being compared can't be converted to an IPAddr object.

# Ruby 2.4>> IPAddr.new("1.2.1.3") == "Some ip address"=> false>> "Some ip address" == IPAddr.new("1.2.1.3")=> false>> IPAddr.new("1.2.1.3") <=> "Some ip address"=> nil>> "Some ip address" <=> IPAddr.new("1.2.1.3")=> nil

This might cause some backward compatibility if our code is expecting theexception which is no longer raised in Ruby 2.4.

In Ruby 2.4, these constants aredeprecated and will be removed infuture version.

# Ruby 2.32.3.1 :001 > TRUE => true2.3.1 :002 > FALSE => false2.3.1 :003 > NIL => nil

# Ruby 2.42.4.0 :001 > TRUE(irb):1: warning: constant ::TRUE is deprecated => true2.4.0 :002 > FALSE(irb):2: warning: constant ::FALSE is deprecated => false2.4.0 :003 > NIL(irb):3: warning: constant ::NIL is deprecated => nil

daily_logger = Logger.new('foo.log', 'daily')weekly_logger = Logger.new('foo.log', 'weekly')monthly_logger = Logger.new('foo.log', 'monthly')

At the end of the specified period, Ruby will change the file extension of thelog file as follows:

foo.log.20170615

The format of the suffix for the rotated log file is %Y%m%d. In Ruby 2.3,there was no way to customize this suffix format.

Ruby 2.4added the abilityto customize the suffix format by passing an extra argumentshift_period_suffix.

# Ruby 2.4logger = Logger.new('foo.log', 'weekly', shift_period_suffix: '%d-%m-%Y')

Now, suffix of the rotated log file will use the custom date format which wepassed.

foo.log.15-06-2017

{ a: 1, b: 2, c: 3 } => { a: 2, b: 4, c: 6 }{ a: "B", c: "D", e: "F" } => { a: "b", c: "d", e: "f" }

We can transform the values of a hash destructively (i.e. modify the originalhash with new values) or non-destructively (i.e. return a new hash instead ofmodifying the original hash).

Prior to Ruby 2.4, we need to use following code to transform the values of ahash.

# Ruby 2.3 Non-destructive version> hash = { a: 1, b: 2, c: 3 } #=> {:a=>1, :b=>2, :c=>3}> hash.inject({}) { |h, (k, v)| h[k] = v * 2; h } #=> {:a=>2, :b=>4, :c=>6}> hash #=> {:a=>1, :b=>2, :c=>3}> hash = { a: "B", c: "D", e: "F" } #=> {:a=>"B", :c=>"D", :e=>"F"}> hash.inject({}) { |h, (k, v)| h[k] = v.downcase; h } #=> {:a=>"b", :c=>"d", :e=>"f"}> hash #=> {:a=>"B", :c=>"D", :e=>"F"}

# Ruby 2.3 Destructive version> hash = { a: 1, b: 2, c: 3 } #=> {:a=>1, :b=>2, :c=>3}> hash.each { |k, v| hash[k] = v * 2 } #=> {:a=>2, :b=>4, :c=>6}> hash #=> {:a=>2, :b=>4, :c=>6}> hash = { a: "B", c: "D", e: "F" } #=> {:a=>"B", :c=>"D", :e=>"F"}> hash.each { |k, v| hash[k] = v.downcase } #=> {:a=>"b", :c=>"d", :e=>"f"}> hash #=> {:a=>"b", :c=>"d", :e=>"f"}

transform_values and transform_values! from Active Support

Active Support has already implemented handy methodsHash#transform_values and Hash#transform_values!to transform hash values.

Now, Ruby 2.4 has also implementedHash#map_v and Hash#map_v!and then renamed toHash#transform_values and Hash#transform_values!for the same purpose.

# Ruby 2.4 Non-destructive version> hash = { a: 1, b: 2, c: 3 } #=> {:a=>1, :b=>2, :c=>3}> hash.transform_values { |v| v * 2 } #=> {:a=>2, :b=>4, :c=>6}> hash #=> {:a=>1, :b=>2, :c=>3}> hash = { a: "B", c: "D", e: "F" } #=> {:a=>"B", :c=>"D", :e=>"F"}> hash.transform_values(&:downcase) #=> {:a=>"b", :c=>"d", :e=>"f"}> hash #=> {:a=>"B", :c=>"D", :e=>"F"}

# Ruby 2.4 Destructive version> hash = { a: 1, b: 2, c: 3 } #=> {:a=>1, :b=>2, :c=>3}> hash.transform_values! { |v| v * 2 } #=> {:a=>2, :b=>4, :c=>6}> hash #=> {:a=>2, :b=>4, :c=>6}> hash = { a: "B", c: "D", e: "F" } #=> {:a=>"B", :c=>"D", :e=>"F"}> hash.transform_values!(&:downcase) #=> {:a=>"b", :c=>"d", :e=>"f"}> hash #=> {:a=>"b", :c=>"d", :e=>"f"}

pry did make our lives easier with the usage ofbinding.pry. However, it was still a bit of an inconvenience to have itinstalled at runtime, while working with the irb.

Ruby 2.4 has now introduced binding.irb. By simply adding binding.irb to ourcode we can open an IRB session.

class ConvolutedProcessdef do_something@variable = 10    binding.irb    # opens a REPL hereendendirb(main):029:0\* ConvolutedProcess.new.do_somethingirb(#<ConvolutedProcess:0x007fc55c827f48>):001:0> @variable=> 10

After creating the logger object we need to set its level.

Ruby 2.3

require 'logger'logger = Logger.new(STDOUT)logger.level = Logger::INFO

If we are working with ActiveRecord::Base.logger, then same code would looksomething like this.

require 'logger'ActiveRecord::Base.logger = Logger.new(STDOUT)ActiveRecord::Base.logger.level = Logger::INFO

As we can see in the both the cases we need to set the level separately afterinstantiating the object.

Ruby 2.4

In Ruby 2.4, level can now be specified in the constructor.

#ruby 2.4require 'logger'logger = Logger.new(STDOUT, level: Logger::INFO)# let's verify itlogger.level      #=> 1

Similarly, other options such as progname, formatter and datetime_format,which prior to Ruby 2.4 had to be explicitly set, can now be set during theinstantiation.

#ruby 2.3require 'logger'logger = Logger.new(STDOUT)logger.level = Logger::INFOlogger.progname = 'bigbinary'logger.datetime_format = '%Y-%m-%d %H:%M:%S'logger.formatter = proc do |severity, datetime, progname, msg|  "#{severity} #{datetime} ==> App: #{progname}, Message: #{msg}\n"endlogger.info("Program started...")#=> INFO 2017-03-16 18:43:58 +0530 ==> App: bigbinary, Message: Program started...

Here is same stuff in Ruby 2.4.

#ruby 2.4require 'logger'logger = Logger.new(STDOUT,  level: Logger::INFO,  progname: 'bigbinary',  datetime_format: '%Y-%m-%d %H:%M:%S',  formatter: proc do |severity, datetime, progname, msg|    "#{severity} #{datetime} ==> App: #{progname}, Message: #{msg}\n"  end)logger.info("Program started...")#=> INFO 2017-03-16 18:47:39 +0530 ==> App: bigbinary, Message: Program started...

Tempfileis used for managing temporary files in Ruby. A Tempfile object creates atemporary file with a unique filename. It behaves just like a File object, andtherefore we can perform all the usual file operations on it.

Why Tempfile when we can use File

These days it is common to store file on services like S3. Let's say that wehave a users.csv file on S3. Working with this file remotely is problematic.In such cases it is desirable to download the file on local machine formanipulation. After the work is done then file should be deleted. Tempfile isideal for such cases.

Basename for tempfile

If we want to create a temporary file then we needed to pass parameter to itprior to Ruby 2.3.

require 'tempfile'file = Tempfile.new('bigbinary')#=> #<Tempfile:/var/folders/jv/fxkfk9_10nb_964rvrszs2540000gn/T/bigbinary-20170304-10828-1w02mqi>

As we can see above the generated file name begins with "bigbinary" word.

Since Tempfile ensures that the generate filename will always be unique thepoint of passing the argument is meaningless. Ruby doc calls this passing"basename".

So in Ruby 2.3.0 it was decided thatthe basename parameter was meaningless for Tempfile#new and an empty stringwill be the default value.

require 'tempfile'file = Tempfile.new#=> #<Tempfile:/var/folders/jv/fxkfk9_10nb_964rvrszs2540000gn/T/20170304-10828-1v855bf>

But the same was not implemented for Tempfile#create.

# Ruby 2.3.0require 'tempfile'Tempfile.create do |f|  f.write "hello"endArgumentError: wrong number of arguments (given 0, expected 1..2)

This was fixed in Ruby 2.4. So nowthe basename parameter for Tempfile.create is set to empty string by default,to keep it consistent with the Tempfile#new method.

# Ruby 2.4require 'tempfile'Tempfile.create do |f|  f.write "hello"end=> 5

Ruby 2.3.x

In the previous versions of Ruby, we could use methods such as floor, ceiland truncate in following ways.

5.54.floor          #=> 55.54.ceil           #=> 65.54.truncate       #=> 5

Providing an argument to these methods would result in ArgumentErrorexception.

Ruby 2.4

Ruby community decided to come up with an option toadd precision argument .

The precision argument, which can be negative, helps us to get result to therequired precision to either side of the decimal point.

The default value for the precision argument is 0.

876.543.floor(-2)       #=> 800876.543.floor(-1)       #=> 870876.543.floor           #=> 876876.543.floor(1)        #=> 876.5876.543.floor(2)        #=> 876.54876.543.ceil(-2)        #=> 900876.543.ceil(-1)        #=> 880876.543.ceil            #=> 877876.543.ceil(1)         #=> 876.6876.543.ceil(2)         #=> 876.55876.543.truncate(-2)    #=> 800876.543.truncate(-1)    #=> 870876.543.truncate        #=> 876876.543.truncate(1)     #=> 876.5876.543.truncate(2)     #=> 876.54

These methods all work the same on Integer as well.

5.floor(2)              #=> 5.05.ceil(2)               #=> 5.05.truncate(2)           #=> 5.0

Let's consider an example of countries that have hosted the Olympics. We onlywant to know when was the first time a country hosted it.

# given object{ 1896 => 'Athens',  1900 => 'Paris',  1904 => 'Chicago',  1906 => 'Athens',  1908 => 'Rome' }# expected outcome{ 1896 => 'Athens',  1900 => 'Paris',  1904 => 'Chicago',  1908 => 'Rome' }

One way to achieve this is to have a collection of unique country names and thencheck if that value is already taken while building the result.

olympics ={ 1896 => 'Athens',  1900 => 'Paris',  1904 => 'Chicago',  1906 => 'Athens',  1908 => 'Rome' }unique_nations = olympics.values.uniqolympics.select{ |year, country| !unique_nations.delete(country).nil? }#=> {1896=>"Athens", 1900=>"Paris", 1904=>"Chicago", 1908=>"Rome"}

As we can see, the above code requires constructing an additional arrayunique_nations.

In processing larger data, loading an array of considerably big size in memoryand then carrying out further processing on it, may result in performance andmemory issues.

In Ruby 2.4, Enumerable class introduces uniqmethod that collects unique elementswhile iterating over the enumerable object.

The usage is similar to that of Array#uniq. Uniqueness can be determined by theelements themselves or by a value yielded by the block passed to the uniqmethod.

olympics = {1896 => 'Athens', 1900 => 'Paris', 1904 => 'Chicago', 1906 => 'Athens', 1908 => 'Rome'}olympics.uniq { |year, country| country }.to_h#=> {1896=>"Athens", 1900=>"Paris", 1904=>"Chicago", 1908=>"Rome"}

Similar method is also implemented in Enumerable::Lazy class. Hence we can nowcall uniq on lazy enumerables.

(1..Float::INFINITY).lazy.uniq { |x| (x**2) % 10 }.first(6)#=> [1, 2, 3, 4, 5, 10]

Ruby also has strip method which is a combination of lstrip and rstrip and canbe used to remove both, leading and trailing whitespaces, from a string.

"    Hello World    ".lstrip    #=> "Hello World    ""    Hello World    ".rstrip    #=> "    Hello World""    Hello World    ".strip     #=> "Hello World"

Prior to Ruby 2.4, the rstrip method was optimized for performance, but thelstrip and strip were somehow missed. In Ruby 2.4, String#lstrip andString#strip methods too have beenoptimized to get the performancebenefit of String#rstrip .

Let's run following snippet in Ruby 2.3 and Ruby 2.4 to benchmark and comparethe performance improvement.

require 'benchmark/ips'Benchmark.ips do |bench|  str1 = " " * 10_000_000 + "hello world" + " " * 10_000_000  str2 = str1.dup  str3 = str1.dup  bench.report('String#lstrip') do    str1.lstrip  end  bench.report('String#rstrip') do    str2.rstrip  end  bench.report('String#strip') do    str3.strip  endend

Result for Ruby 2.3

Warming up --------------------------------------       String#lstrip     1.000  i/100ms       String#rstrip     8.000  i/100ms        String#strip     1.000  i/100msCalculating -------------------------------------       String#lstrip     10.989  ( 0.0%) i/s -     55.000  in   5.010903s       String#rstrip     92.514  ( 5.4%) i/s -    464.000  in   5.032208s        String#strip     10.170  ( 0.0%) i/s -     51.000  in   5.022118s

Result for Ruby 2.4

Warming up --------------------------------------       String#lstrip    14.000  i/100ms       String#rstrip     8.000  i/100ms        String#strip     6.000  i/100msCalculating -------------------------------------       String#lstrip    143.424  ( 4.2%) i/s -    728.000  in   5.085311s       String#rstrip     89.150  ( 5.6%) i/s -    448.000  in   5.041301s        String#strip     67.834  ( 4.4%) i/s -    342.000  in   5.051584s

From the above results, we can see that in Ruby 2.4, String#lstrip is around14x faster while String#strip is around 6x faster. String#rstrip asexpected, has nearly the same performance as it was already optimized inprevious versions.

Performance remains same for multi-byte strings

Strings can have single byte or multi-byte characters.

For example L Hello World is a multi-byte string because of the presence of which is a multi-byte character.

'e'.bytesize        #=> 1''.bytesize        #=> 2

Let's do performance benchmarking with string L hello world instead ofhello world.

Result for Ruby 2.3

Warming up --------------------------------------       String#lstrip     1.000  i/100ms       String#rstrip     1.000  i/100ms        String#strip     1.000  i/100msCalculating -------------------------------------       String#lstrip     11.147  ( 9.0%) i/s -     56.000  in   5.034363s       String#rstrip      8.693  ( 0.0%) i/s -     44.000  in   5.075011s        String#strip      5.020  ( 0.0%) i/s -     26.000  in   5.183517s

Result for Ruby 2.4

Warming up --------------------------------------       String#lstrip     1.000  i/100ms       String#rstrip     1.000  i/100ms        String#strip     1.000  i/100msCalculating -------------------------------------       String#lstrip     10.691  ( 0.0%) i/s -     54.000  in   5.055101s       String#rstrip      9.524  ( 0.0%) i/s -     48.000  in   5.052678s        String#strip      4.860  ( 0.0%) i/s -     25.000  in   5.152804s

As we can see, the performance for multi-byte strings is almost the same acrossRuby 2.3 and Ruby 2.4.

Explanation

The optimization introduced is related to how the strings are parsed to detectfor whitespaces. Checking for whitespaces in multi-byte string requires anadditional overhead. So the patchadds an initial condition to check if the string is a single byte string, and ifso, processes it separately.

In most of the cases, the strings are single byte so the performance improvementwould be visible and helpful.

# lotr.txtThree Rings for the Elven-kings under the sky,Seven for the Dwarf-lords in their halls of stone,Nine for Mortal Men doomed to die,One for the Dark Lord on his dark throneIn the Land of Mordor where the Shadows lie.

Ruby 2.3

IO.readlines('lotr.txt')#=> ["Three Rings for the Elven-kings under the sky,\n", "Seven for the Dwarf-lords in their halls of stone,\n", "Nine for Mortal Men doomed to die,\n", "One for the Dark Lord on his dark throne\n", "In the Land of Mordor where the Shadows lie."]

As we can see, the lines in the array have a \n, newline character, which isnot skipped while reading the lines. The newline character needs to be choppedin most of the cases. Prior to Ruby 2.4, it could be done in the following way.

IO.readlines('lotr.txt').map(&:chomp)#=> ["Three Rings for the Elven-kings under the sky,", "Seven for the Dwarf-lords in their halls of stone,", "Nine for Mortal Men doomed to die,", "One for the Dark Lord on his dark throne", "In the Land of Mordor where the Shadows lie."]

Ruby 2.4

Since it was a common requirement, Ruby team decided toadd an optional parameter to thereadlines method. So the same can now be achieved in Ruby 2.4 in the followingway.

IO.readlines('lotr.txt', chomp: true)#=> ["Three Rings for the Elven-kings under the sky,", "Seven for the Dwarf-lords in their halls of stone,", "Nine for Mortal Men doomed to die,", "One for the Dark Lord on his dark throne", "In the Land of Mordor where the Shadows lie."]

Additionally, IO#gets, IO#readline, IO#each_line, IO#foreach methodsalso have been modified to accept an optional chomp flag.

> require 'open-uri'> open('http://www.google.com/gmail')RuntimeError: redirection forbidden: http://www.google.com/gmail -> https://www.google.com/gmail/

To get around this issue, we could useopen_uri_redirectionsgem.

> require 'open-uri'> require 'open_uri_redirections'> open('http://www.google.com/gmail/', :allow_redirections => :safe)=> #<Tempfile:/var/folders/jv/fxkfk9_10nb_964rvrszs2540000gn/T/open-uri20170228-41042-2fffoa>

Ruby 2.4

In Ruby 2.4, this issue is fixed. Sonow http to https redirection is possible using open-uri.

> require 'open-uri'> open('http://www.google.com/gmail')=> #<Tempfile:/var/folders/jv/fxkfk9_10nb_964rvrszs2540000gn/T/open-uri20170228-41077-1bkm1dv>

Note that redirection from https to http will raise an error, like it did inprevious versions, since that has possible security concerns.

Dir.entries("/usr/lib").size == 2       #=> falseDir.entries("/home").size == 2          #=> true

Every directory in Unix filesystem contains at least two entries. These are.(current directory) and ..(parent directory).

Hence, the code above checks if there are only two entries and if so, consider adirectory empty.

Again, this code only works for UNIX filesystems and fails on Windows machines,as Windows directories don't have . or ...

Dir.empty?

Considering all this, Ruby has finallyincluded a new method Dir.empty?that takes directory path as argument and returns boolean as an answer.

Here is an example.

Dir.empty?('/Users/rtdp/Documents/posts')   #=> true

Most importantly this method works correctly in all platforms.

File.empty?

To check if a file is empty, Ruby has File.zero? method. This checks if thefile exists and has zero size.

File.zero?('/Users/rtdp/Documents/todo.txt')    #=> true

After introducing Dir.empty? it makes sense toadd File.empty? as an alias toFile.zero?

File.empty?('/Users/rtdp/Documents/todo.txt')    #=> true

567321.digits#=> [1, 2, 3, 7, 6, 5]567321.digits[3]#=> 7

We can also supply a different base as an argument.

0123.digits(8)#=> [3, 2, 1]0xabcdef.digits(16)#=> [15, 14, 13, 12, 11, 10]

Use case of digits

We can use Integer#digits to sum all the digits in an integer.

123.to_s.chars.map(&:to_i).sum#=> 6123.digits.sum#=> 6

Also while calculating checksums likeLuhn andVerhoeff, Integer#digitswill help in reducing string allocation.

For example:

str1 = "Sample string"str2 = str1.dupstr1.eql?(str2) #=> truestr1.equal?(str2) #=> false

We can see that object ids of the objects are not same.

str1.object_id #=> 70334175057920str2.object_id #=> 70334195702480

In ruby, Set doesnot allow duplicate items in its collection. To determine if two items are equalor not in a Set ruby uses Object#eql? and not Object#equal?.

So if we want to add two different objects with the same values in a set, thatwould not have been possible prior to Ruby 2.4 .

Ruby 2.3

require 'set'set = Set.new #=> #<Set: {}>str1 = "Sample string" #=> "Sample string"str2 = str1.dup #=> "Sample string"set.add(str1) #=> #<Set: {"Sample string"}>set.add(str2) #=> #<Set: {"Sample string"}>

But with the newSet#compare_by_identity method introduced in Ruby 2.4,sets can now compare its values using Object#equal? and check for the exactsame objects.

Ruby 2.4

require 'set'set = Set.new.compare_by_identity #=> #<Set: {}>str1 = "Sample string" #=> "Sample string"str2 = str1.dup #=> "Sample string"set.add(str1) #=> #<Set: {"Sample string"}>set.add(str2) #=> #<Set: {"Sample string", "Sample string"}>

Set#compare_by_identity?

Ruby 2.4 also provides thecompare_by_identity? methodto know if the set will compare its elements by their identity.

require 'set'set1= Set.new #=> #<Set: {}>set2= Set.new.compare_by_identity #=> #<Set: {}>set1.compare_by_identity? #=> falseset2.compare_by_identity? #=> true

We can use MatchData#[] to extract named capture and positional capturegroups.

pattern=/(?<number>\d+) (?<word>\w+)/pattern.match('100 thousand')[:number]#=> "100"pattern=/(\d+) (\w+)/pattern.match('100 thousand')[2]#=> "thousand"

Positional capture groups could also be extracted using MatchData#values_at.

pattern=/(\d+) (\w+)/pattern.match('100 thousand').values_at(2)#=> ["thousand"]

Changes in Ruby 2.4

In Ruby 2.4, we can pass string or symbol to extract named capture groups tomethod #values_at.

pattern=/(?<number>\d+) (?<word>\w+)/pattern.match('100 thousand').values_at(:number)#=> ["100"]

Prior to Ruby 2.4, Float and BigDecimal responded to methods infinite? andfinite?, whereas Fixnum and Bignum did not.

Ruby 2.3

#infinite?5.0.infinite?=> nilFloat::INFINITY.infinite?=> 15.infinite?NoMethodError: undefined method `infinite?' for 5:Fixnum

#finite?5.0.finite?=> true5.finite?NoMethodError: undefined method `finite?' for 5:Fixnum

Ruby 2.4

To make behavior for all the numeric values to be consistent,infinite? and finite? were added to Fixnum and Bignumeven though they would always return nil.

This gives us ability to call these methods irrespective of whether they aresimple numbers or floating numbers.

#infinite?5.0.infinite?=> nilFloat::INFINITY.infinite?=> 15.infinite?=> nil

#finite?5.0.finite?=> true5.finite?=> true

clamp method takes min and max as two arguments to define the range of valuesin which the given argument should be clamped.

Clamping numbers

clamp can be used to keep a number within the range of min, max.

10.clamp(5, 20)=> 1010.clamp(15, 20)=> 1510.clamp(0, 5)=> 5

Clamping strings

Similarly, strings can also be clamped within a range.

"e".clamp("a", "s")=> "e""e".clamp("f", "s")=> "f""e".clamp("a", "c")=> "c""this".clamp("thief", "thin")=> "thin"

Internally, this method relies on applying thespaceship <=> operatorbetween the object and the min & max arguments.

if x <=> min < 0, x = min;if x <=> max > 0 , x = maxelse x

According to RFC 4180, we cannothave unescaped double quotes in CSV input since such data can't be parsed.

We get MalformedCSVError error when the CSV data does not conform to RFC 4180.

Ruby 2.4 has added aliberal parsing option to parse suchbad data. When it is set to true, Ruby will try to parse the data even whenthe data does not conform to RFC 4180.

# Before Ruby 2.4> CSV.parse_line('one,two",three,four')CSV::MalformedCSVError: Illegal quoting in line 1.# With Ruby 2.4> CSV.parse_line('one,two",three,four', liberal_parsing: true)=> ["one", "two\"", "three", "four"]

[1, 4, 7, 10, 2, 6, 15].chunk { |item| item > 5 }.each { |values| p values }=> [false, [1, 4]][true, [7, 10]][false, [2]][true, [6, 15]]

Prior to Ruby 2.4, passing a block to chunk method was must.

array = [1,2,3,4,5,6]array.chunk=> ArgumentError: no block given

Enumerable#chunk without block in Ruby 2.4

In Ruby 2.4, we will be able to usechunk without passing block. It justreturns the enumerator object which we can use to chain further operations.

array = [1,2,3,4,5,6]array.chunk=> <Enumerator: [1, 2, 3, 4, 5, 6]:chunk>

Reasons for this change

Let's take thecase of listingconsecutive integers in an array of ranges.

# Before Ruby 2.4integers = [1,2,4,5,6,7,9,13]integers.enum_for(:chunk).with_index { |x, idx| x - idx }.map do |diff, group|  [group.first, group.last]end=> [[1,2],[4,7],[9,9],[13,13]]

We had to useenum_for here aschunk can't be called without block.

enum_for creates a new enumerator object which will enumerate by calling themethod passed to it. In this case the method passed was chunk.

With Ruby 2.4, we can use chunk method directly without using enum_for as itdoes not require a block to be passed.

# Ruby 2.4integers = [1,2,4,5,6,7,9,13]integers.chunk.with_index { |x, idx| x - idx }.map do |diff, group|  [group.first, group.last]end=> [[1,2],[4,7],[9,9],[13,13]]

# Before Ruby 2.41.class         #=> Fixnum(2 ** 62).class #=> Bignum

In general routine work we don't have to worry about whether the number we aredealing with is Bignum or Fixnum. It's just an implementation detail.

Interestingly, Ruby also has Integer class which is superclass for Fixnumand Bignum.

Starting with Ruby 2.4, Fixnum and Bignumare unified into Integer.

# Ruby 2.41.class         #=> Integer(2 ** 62).class #=> Integer

Starting with Ruby 2.4 usage of Fixnum and Bignum constantsis deprecated.

# Ruby 2.4>> Fixnum(irb):6: warning: constant ::Fixnum is deprecated=> Integer>> Bignum(irb):7: warning: constant ::Bignum is deprecated=> Integer

How to know if a number is Fixnum, Bignum or Integer?

We don't have to worry about this change most of the times in our applicationcode. But libraries like Rails use the class of numbers for taking certaindecisions. These libraries need to support both Ruby 2.4 and previous versionsof Ruby.

Easiest way to know whether the Ruby version is using integer unification or notis to check class of 1.

# Ruby 2.41.class #=> Integer# Before Ruby 2.41.class #=> Fixnum

Look at PR #25056 to see how Railsis handling this case.

Similarly Arel isalso supportingboth Ruby 2.4 and previous versions of Ruby.

(1..10).to_a.max#=> 10(1..10).to_a.method(:max)#=> #<Method: Array(Enumerable)#max>

Ruby 2.4 adds Array#min and Array#maxwhich are much faster than Enumerable#max and Enuermable#min.

Following benchmark is based onhttps://blog.blockscore.com/new-features-in-ruby-2-4.

Benchmark.ips do |bench|  NUM1 = 1_000_000.times.map { rand }  NUM2 = NUM1.dup  ENUM_MAX = Enumerable.instance_method(:max).bind(NUM1)  ARRAY_MAX = Array.instance_method(:max).bind(NUM2)  bench.report('Enumerable#max') do    ENUM_MAX.call  end  bench.report('Array#max') do    ARRAY_MAX.call  end  bench.compare!endWarming up --------------------------------------      Enumerable#max     1.000  i/100ms           Array#max     2.000  i/100msCalculating -------------------------------------      Enumerable#max     17.569  ( 5.7%) i/s -     88.000  in   5.026996s           Array#max     26.703  ( 3.7%) i/s -    134.000  in   5.032562sComparison:           Array#max:       26.7 i/s      Enumerable#max:       17.6 i/s - 1.52x  slowerBenchmark.ips do |bench|  NUM1 = 1_000_000.times.map { rand }  NUM2 = NUM1.dup  ENUM_MIN = Enumerable.instance_method(:min).bind(NUM1)  ARRAY_MIN = Array.instance_method(:min).bind(NUM2)  bench.report('Enumerable#min') do    ENUM_MIN.call  end  bench.report('Array#min') do    ARRAY_MIN.call  end  bench.compare!endWarming up --------------------------------------      Enumerable#min     1.000  i/100ms           Array#min     2.000  i/100msCalculating -------------------------------------      Enumerable#min     18.621  ( 5.4%) i/s -     93.000  in   5.007244s           Array#min     26.902  ( 3.7%) i/s -    136.000  in   5.064815sComparison:           Array#min:       26.9 i/s      Enumerable#min:       18.6 i/s - 1.44x  slower

This benchmark shows that the new methods Array#max and Array#min are about1.5 times faster than Enumerable#max and Enumerable#min.

Similar to Enumerable#max and Enumerable#min, Array#max and Array#minalso assumes that the objects useComparable mixin to definespaceship <=> operator for comparing the elements.

We use shoryuken to process SQS messages inside of docker containers. A whileback we noticed that memory was growing without bound. After every few days, wehad to restart all the docker containers as a temporary workaround.

Since the workers were inside of a docker container we had limited tools.So we went ahead with the UNIX way of investigating the issue.

First we noticed that the number of threads inside the worker was high, 115 inour case. shoryuken boots up all the worker threads atstartup.

# ps --no-header uH p <PID> | wc -l#=> 115

The proc filesystem exposes a lot of useful information of all the runningprocesses. ``/proc/[pid]/task` directory has information about all the threadsof a process.

Some of the threads with lower ID's were executing syscall 23(select) and271 (ppoll).These threads were waitingfor a message to arrive in the SQS queue, but most of the threads wereexecuting syscall 202(futex).

At this point we had an idea about the root cause of the memory leak -it was due to the workerstarting a lot of threads which were not getting terminated.We wanted to knowhow and when these threads are started.

Ruby 2.0.0 introduced tracepoint,which provides an interface to a lot ofinternal ruby events like when a exception is raised, when a method is called orwhen a method returns, etc.

We added the following code to our workers.

tc = TracePoint.new(:thread_begin, :thread_end) do |tp|  puts tp.event  puts tp.self.classendtc.enable

Executing the ruby workers with tracing enabled revealed that a newCelluloid::Thread was being created before each method was processed and thatthread was never terminated. Hence the number of zombie threads in the workerwas growing with the number messages processed.

thread_beginCelluloid::Thread[development] [306203a5-3c07-4174-b974-77390e8a4fc3] SQS Message: ...snip...thread_beginCelluloid::Thread[development] [2ce2ed3b-d314-46f1-895a-f1468a8db71e] SQS Message: ...snip...

Unfortunately tracepoint didn't pinpoint the place where the thread was started,hence we added a couple of puts statements to investigate the issue further.

After a lot of debugging, we were able to find that a new thread was started toincrease the visibility timeof the SQS message in a shoryuken middleware whenauto_visibility_timeout was true.

The fix was toterminatethe thread after the work is done.

It has methods #names and #captures to return the names used for capturingand the actual captured data respectively.

pattern = /(?<number>\d+) (?<word>\w+)/match_data = pattern.match('100 thousand')#=> #<MatchData "100 thousand" number:"100" word:"thousand">>> match_data.names=> ["number", "word"]>> match_data.captures=> ["100", "thousand"]

If we want all named captures in a key value pair, we have to combine the resultof names and captures.

match_data.names.zip(match_data.captures).to_h#=> {"number"=>"100", "word"=>"thousand"}

Ruby 2.4 adds #named_captures whichreturns both the name and data of the capture groups.

pattern=/(?<number>\d+) (?<word>\w+)/match_data = pattern.match('100 thousand')match_data.named_captures#=> {"number"=>"100", "word"=>"thousand"}

Regexp#===

It returns true/false and sets the $~ global variable.

/stat/ === "case statements"#=> true$~#=> #<MatchData "stat">

Regexp#=~

It returns integer position it matched or nil if no match. It also sets the $~global variable.

/stat/ =~ "case statements"#=> 5$~#=> #<MatchData "stat">

Regexp#match

It returns match data and also sets the $~ global variable.

/stat/.match("case statements")#=> #<MatchData "stat">$~#=> #<MatchData "stat">

Ruby 2.4 adds Regexp#match?

This new method just returnstrue/false and does not set any global variables.

/case/.match?("case statements")#=> true

So Regexp#match? is good option when we are only concerned with the fact thatregex matches or not.

Regexp#match? is also faster than its counterparts as it reduces objectallocation by not creating a back reference and changing $~.

require 'benchmark/ips'Benchmark.ips do |bench|  EMAIL_ADDR = 'disposable.style.email.with+symbol@example.com'  EMAIL_REGEXP_DEVISE = /\A[^@\s]+@([^@\s]+\.)+[^@\W]+\z/  bench.report('Regexp#===') do    EMAIL_REGEXP_DEVISE === EMAIL_ADDR  end  bench.report('Regexp#=~') do    EMAIL_REGEXP_DEVISE =~ EMAIL_ADDR  end  bench.report('Regexp#match') do    EMAIL_REGEXP_DEVISE.match(EMAIL_ADDR)  end  bench.report('Regexp#match?') do    EMAIL_REGEXP_DEVISE.match?(EMAIL_ADDR)  end  bench.compare!end#=> Warming up --------------------------------------#=>          Regexp#===   103.876k i/100ms#=>           Regexp#=~   105.843k i/100ms#=>        Regexp#match    58.980k i/100ms#=>       Regexp#match?   107.287k i/100ms#=> Calculating -------------------------------------#=>          Regexp#===      1.335M ( 9.5%) i/s -      6.648M in   5.038568s#=>           Regexp#=~      1.369M ( 6.7%) i/s -      6.880M in   5.049481s#=>        Regexp#match    709.152k ( 5.4%) i/s -      3.539M in   5.005514s#=>       Regexp#match?      1.543M ( 4.6%) i/s -      7.725M in   5.018696s#=>#=> Comparison:#=>       Regexp#match?:  1542589.9 i/s#=>           Regexp#=~:  1369421.3 i/s - 1.13x  slower#=>          Regexp#===:  1335450.3 i/s - 1.16x  slower#=>        Regexp#match:   709151.7 i/s - 2.18x  slower

[1, 2, 3, 4] => 10{a: 1, b: 6, c: -3} => 4

Active Support already implementsEnumerable#sum

> [1, 2, 3, 4].sum #=> 10> {a: 1, b: 6, c: -3}.sum{ |k, v| v**2 } #=> 46> ['foo', 'bar'].sum # concatenation of strings #=> "foobar"> [[1], ['abc'], [6, 'qwe']].sum # concatenation of arrays #=> [1, "abc", 6, "qwe"]

Until Ruby 2.3, we had to use Active Support to use Enumerable#sum method orwe could use #inject which isused by Active Support under the hood.

Ruby 2.4.0 implementsEnumerable#sumas part of the language itself.

Let's take a look at how sum method fares on some of the enumerable objects inRuby 2.4.

> [1, 2, 3, 4].sum #=> 10> {a: 1, b: 6, c: -3}.sum { |k, v| v**2 } #=> 46> ['foo', 'bar'].sum #=> TypeError: String can't be coerced into Integer> [[1], ['abc'], [6, 'qwe']].sum #=> TypeError: Array can't be coerced into Integer

As we can see, the behavior of Enumerable#sum from Ruby 2.4 is same as that ofActive Support in case of numbers but not the same in case of string or arrayconcatenation. Let's see what is the difference and how we can make it work inRuby 2.4 as well.

Understanding addition/concatenation identity

The Enumerable#sum method takes an optional argument which acts as anaccumulator. Both Active Support and Ruby 2.4 accept this argument.

When identity argument is not passed, 0 is used as default accumulator in Ruby2.4 whereas Active Support uses nil as default accumulator.

Hence in the cases of string and array concatenation, the error occurred in Rubybecause the code attempts to add a string and array respectively to 0.

To overcome this, we need to pass proper addition/concatenation identity as anargument to the sum method.

The addition/concatenation identity of an object can be defined as the valuewith which calling + operation on an object returns the same object.

> ['foo', 'bar'].sum('') #=> "foobar"> [[1], ['abc'], [6, 'qwe']].sum([]) #=> [1, "abc", 6, "qwe"]

What about Rails ?

As we have seen earlier, Ruby 2.4 implements Enumerable#sum favouring numericoperations whereas also supporting non-numeric callers with the identityelement. This behavior is not entirely same as that of Active Support. But stillActive Support can make use of the native sum method whenever possible. There isalready a pull request open whichuses Enumerable#sum from Ruby whenever possible. This will help gain someperformance boost as the Ruby's method is implemented natively in C whereas thatin Active Support is implemented in Ruby.

Ruby 2.3

String#concat and Array#concat

string = "Good"string.concat(" morning")#=> "Good morning"array = ['a', 'b', 'c']array.concat(['d'])#=> ["a", "b", "c", "d"]

String#prepend

string = "Morning"string.prepend("Good ")#=> "Good morning"

Before Ruby 2.4, we could pass only one argument to these methods. So we couldnot add multiple items in one shot.

string = "Good"string.concat(" morning", " to", " you")#=> ArgumentError: wrong number of arguments (given 3, expected 1)

Changes with Ruby 2.4

In Ruby 2.4, we can pass multiple arguments and Ruby processes each argument oneby one.

String#concat and Array#concat

string = "Good"string.concat(" morning", " to", " you")#=> "Good morning to you"array = ['a', 'b']array.concat(['c'], ['d'])#=> ["a", "b", "c", "d"]

String#prepend

string = "you"string.prepend("Good ", "morning ", "to ")#=> "Good morning to you"

These methods work even when no argument is passed unlike in previous versionsof Ruby.

"Good".concat#=> "Good"

Difference between concat and shovel << operator

Though shovel << operator can be used interchangeably with concat when weare calling it once, there is a difference in the behavior when calling itmultiple times.

str = "Ruby"str << strstr#=> "RubyRuby"str = "Ruby"str.concat strstr#=> "RubyRuby"str = "Ruby"str << str << str#=> "RubyRubyRubyRuby"str = "Ruby"str.concat str, strstr#=> "RubyRubyRuby"

So concat behaves as appending present content to the caller twice. Whereascalling << twice is just sequence of binary operations. So the argument forthe second call is output of the first << operation.

{ "name" => "prathamesh", "email" => nil} => { "name" => "prathamesh" }

Active Support already has a solution for this in the form ofHash#compactandHash#compact!.

hash = { "name" => "prathamesh", "email" => nil}hash.compact #=> { "name" => "prathamesh" }hash #=> { "name" => "prathamesh", "email" => nil}hash.compact! #=> { "name" => "prathamesh" }hash #=> { "name" => "prathamesh" }

Now, Ruby 2.4 will have these 2 methods in thelanguageitself, so even those not using Railsor Active Support will be able to use them. Additionally it will also giveperformance boost over the Active Support versions because now these methods areimplemented in C natively whereas the Active Support versions are in Ruby.

There is already apull request open in Rails to usethe native versions of these methods from Ruby 2.4 whenever available so that wewill be able to use the performance boost.

CircleCI uses circle.yml file for configuration. Before configuring JRuby, this is how it looked like:

machine:  ruby:    version: 2.1.5dependencies:  pre:    - ./bundle_install_circle_ci.sh  cache_directories:    - "~/vendor/bundle"test:  override:    - ? bundle exec rspec --format progress --format documentation --format RspecJunitFormatter --out $CIRCLE_TEST_REPORTS/app.xml      : parallel: true        pwd: app        files:          - spec/**/*_spec.rb

<br/>Here are the steps to enable JRuby in CircleCI.

Specify the JDK version

We need to specify a JDK version before using JRuby.

machine:  java:    version: openjdk7

Install proper dependencies

We needed to use JRuby 9.0.4.0 but the version of JRuby that came with Ubuntu 12.04 image of CircleCI was different. We added rvm install command as follows to install specific version that we wanted. Also we can configure any script (like bundle install) that needs to run before running tests.

dependencies:  pre:    - rvm install jruby-9.0.4.0    - ./bundle_install_jruby_circle_ci.sh  cache_directories:    - "~/vendor/bundle"

Configure JRuby

We used rvm-exec to set JRuby for running tests for this particular component in the test section. Otherwise by default it picks up MRI.

test:  override:    - ? rvm-exec jruby-9.0.4.0 bash -c "bundle exec rspec --format progress --format documentation --format RspecJunitFormatter --out $CIRCLE_TEST_REPORTS/app_jruby.xml"      : parallel: true        pwd: app        files:          - spec/**/*_spec.rb

Improving test runs on JRuby

Once we started running tests with JRuby, we observed it was taking comparatively slower to finish all tests. Most of the time was spent in starting the JVM. We made it faster by setting --dev parameter in JRUBY_OPTS environment variable. This parameter improves JRuby boot time and it shaved more than a minute time for us.

machine:  environment:    JRUBY_OPTS: "--dev"

Here is the final circle.yml file:

# circle.ymlmachine:  ruby:    version: 2.1.5  java:    version: openjdk7  environment:    JRUBY_OPTS: "--dev"dependencies:  pre:    - rvm install jruby-9.0.4.0    - ./bundle_install_jruby_circle_ci.sh  cache_directories:    - "~/vendor/bundle"test:  override:    - ? rvm-exec jruby-9.0.4.0 bash -c "bundle exec rspec --format progress --format documentation --format RspecJunitFormatter --out $CIRCLE_TEST_REPORTS/app_jruby.xml"      : parallel: true        pwd: app        files:          - spec/**/*_spec.rb    - ? bundle exec rspec --format progress --format documentation --format RspecJunitFormatter --out $CIRCLE_TEST_REPORTS/app_mri.xml      : parallel: true        pwd: app        files:          - spec/**/*_spec.rb

We create a directory called _data in the root folder. This directory has asingle file called authors.yml which in our case looks like this.

vipulnsward:  name: Vipul  avatar: http://bigbinary.com/assets/team/vipul.jpg  github: vipulnsward  twitter: vipulnswardneerajsingh0101:  name: Neeraj Singh  avatar: http://bigbinary.com/assets/team/neeraj.jpg  github: neerajsingh0101  twitter: neerajsingh0101

We do not need to do anything to load authors.yml . It is automaticallyloaded by jekyll.

When we create a blog then the top of the blog looks like this.

---layout: posttitle: How to deploy jekyll site to herokucategories: [Ruby]author_github: neerajsingh0101---

Notice the last line where we have put in the author's github id. That's theidentifier we use to pull in author's information.

In order to display author's name we have following code in the layout.

{% raw %}<span class="author-name">  {{ site.data.authors[page.author_github].name }}</span>{% endraw %}

Similarly to display author's twitter handle and github id we have followingcode.

{% raw %}<a href="www.twitter.com/{{site.data.authors[page.author_github].twitter}}">  <i class="ico-twitter"></i></a><a href="www.github.com/{{site.data.authors[page.author_github].github}}">  <i class="ico-github"></i></a>{% endraw %}

Now the blog will display the author information and all this information isnicely centralized in one single file.

Bug in ruby

Let's try a small program.

class Labdef dayputs 'invoked''sunday'enddef runday = dayendendputs Lab.new.run

What do you think would be printed on your terminal when you run the above program.

If you are using ruby 2.1 or below then you will see nothing. Why is that ? That's because of a bug in ruby.

This is bug number 9593 in ruby issue tracker.

In the statement day = day the left hand side variable assignment is stopping the call to method day. So the method day is never invoked.

Another variation of the same bug

class Labdef dayputs 'invoked''sunday'enddef run( day: day)endendputs Lab.new.run

In the above case we are using the keyword argument feature added in Ruby 2.0 . If you are unfamiliar with keyword arguments feature of ruby then checkout this excellent video by Peter Cooper.

In this case again the same behavior is exhibited. The method day is never invoked.

How this bug affects Rails community

You might be thinking that I would never write code like that. Why would you have a variable name same as method name.

Well Rails had this bug because rails has code like this.

def has_cached_counter?(reflection = reflection)end

In this case method reflection never got called and the variable reflection was always assigned nil.

Fixing the bug

Nobu fixed this bug in ruby 2.2.0. By the way Nobu is also known as "ruby patch-monster" because of amount of patches he applies to ruby.

So this bug is fixed in ruby 2.2.0. What about the people who are not using ruby 2.2.0.

The simple solution is not to omit the parameter. If we change the above code to

def has_cached_counter?(reflection = reflection())end

then we are explicitly invoking the method reflection and the variable reflection will be assigned the output of method reflection.

And this is how Aaron Patterson fixed six years old bug.

Add exclude vendor to _config.yml

Open _config.yml and add following line at the very bottom.

exclude: ['vendor']

Add Procfile

Create a new file called Procfile at the root of the project with followingcontent.

web: bundle exec jekyll build && bundle exec thin start -p\$PORT -Vconsole: echo consolerake: echo rake

Add Gemfile

Add Gemfile at the root of the project.

source 'https://rubygems.org'gem 'jekyll', '2.4.0'gem 'rake'gem 'foreman'gem 'thin'gem 'rack-contrib'

Add config.ru

Add config.ru at the root of the project with following content.

require 'rack/contrib/try_static'use Rack::TryStatic,:root => "\_site",:urls => %w[/],:try => ['.html', 'index.html', '/index.html']run lambda { |env|return [404, {'Content-Type' => 'text/html'}, ['Not Found']]}

Test on local machine first

Test locally by executing bundle exec jekyll serve.

Push code to heroku

Now run bundle install and add the Gemfile.lock to the repository and pushthe repository to heroku.

What is the issue

In the referred blog we are trying to find if --force or -f argument waspassed to the git push command.

The kernel knows the arguments that was passed to the command. So the only wayto find that answer would be to ask kernel what was the full command. The toolto deal with such issues is ps.

In order to play with ps command let's write a simple ruby program first.

# sl.rbputs Process.pidputs Process.ppidsleep 99999999

In terminal execute ruby sl.rb. In another terminal execute ps.

$ ps  PID TTY           TIME CMD82246 ttys000    0:00.51 -bash87070 ttys000    0:00.04 ruby loop.rb a, b, c82455 ttys001    0:00.40 -bash

So here I have two bash shell open in two different tabs in my terminal. Firstterminal tab is running s1.rb. The second terminal tab is running ps. In thesecond terminal we can see the arguments that were passed to program s1.

By default ps lists all the processes belonging to the user executing thecommand and the processes started from the current terminal.

Option -p

ps -p87070 would show result only for the given process id.

$ ps -p 87070  PID TTY           TIME CMD87070 ttys000    0:00.04 ruby loop.rb a, b, c

We can pass more than on process id.

$ ps -o pid,command -p87070,82246  PID COMMAND82246 -bash87070 ruby loop.rb a, b, c

Option -o

ps -o can be used to select the attributes that we want to be shown. Forexample I want only pids to be shown.

$ ps -o pid  PID822468707082455

Now I want pid and command.

$ ps -o pid,command  PID COMMAND82246 -bash87070 ruby loop.rb a, b, c82455 -bash

I want result only for a certain process id.

$ ps -o command -p87070COMMANDruby loop.rb a, b, c

Now we have the arguments that were passed to the command. This is the code thatarticle was talking about.

For the sake of completeness let's see a few more options.

Option -e

ps -e would list all processes.

$ ps -e  PID TTY           TIME CMD    1 ??         2:56.20 /sbin/launchd   11 ??         0:01.90 /usr/libexec/UserEventAgent (System)   12 ??         0:02.11 /usr/libexec/kextd   14 ??         0:09.00 /usr/sbin/notifyd   15 ??         0:05.81 /usr/sbin/securityd -i   ........................................   ........................................

Option -f

ps -f would list a lot more attributes including ppid.

$ ps -f  UID   PID  PPID   C STIME   TTY           TIME CMD  501 82246 82245   0  2:06PM ttys000    0:00.51 -bash  501 87070 82246   0  4:54PM ttys000    0:00.04 ruby loop.rb a, b, c  501 82455 82452   0  2:07PM ttys001    0:00.42 -bash

Parent process id is ppid

We know that every process has a process id. This is usually referred as pid.In *nix world every process has a parent process. And in ruby the way to getthe "process id" of the parent process is through ppid.

Let's see it in action. Time to fire up irb.

irb(main):002:0> Process.pid=> 83132irb(main):003:0> Process.ppid=> 82455

Now keep the irb session open and go to anther terminal tab. In this new tabexecute pstree -p 83132

$ pstree -p 83132-+= 00001 root /sbin/launchd \-+= 00151 nsingh /sbin/launchd   \-+= 00189 nsingh /Applications/Utilities/Terminal.app/Contents/MacOS/Terminal -psn_0_45067     \-+= 82452 root login -pf nsingh       \-+= 82455 nsingh -bash         \--= 83132 nsingh irb

If pstree is not available then you can easily install it usingbrew install pstree.

As you can see from the output the process id 83132 is at the very bottom of thetree. The parent process id is 82455 which belongs to "bash shell".

In irb session when we did Process.ppid then we got the same value 82455.

First let's add a UIViewController that is initialized with an image.

class ImageViewController < UIViewController  def initWithImage(image)    @image = image  endend

UIScrollView and UIImageView

Now, we will add a UIScrollView with frame size set to full screen size andsome other properties as listed below.

scrollView = UIScrollView.alloc.initWithFrame(UIScreen.mainScreen.bounds)scrollView.scrollEnabled = falsescrollView.clipsToBounds = truescrollView.contentSize = @image.sizescrollView.minimumZoomScale = 1.0scrollView.maximumZoomScale = 4.0scrollView.zoomScale = 0.3

Create a new UIImageView and add it to the scrollView created above.

imageView = UIImageView.alloc.initWithImage(@image)imageView.contentMode = UIViewContentModeScaleAspectFitimageView.userInteractionEnabled = trueimageView.frame = scrollView.bounds

We are setting the image view's content mode toUIViewContentModeScaleAspectFit. Content mode can be set to eitherUIViewContentModeScaleToFill, UIViewContentModeAspectFill orUIViewContentModeScaleAspectFit depending on what suits your app. By default,contentMode property for most views is set to UIViewContentModeScaleToFill,which causes the views contents to be scaled to fit the new frame size.This Apple docexplains this behavior.

We need to add the above imageView as a subview to our scrollView.

scrollView.addSubview(imageView)self.view.addSubview(@scrollView)

This is how our controller looks with all the above additions.

class ImageViewController < UIViewController  def initWithImage(image)    @image = image    scrollView = UIScrollView.alloc.initWithFrame(UIScreen.mainScreen.bounds)    scrollView.scrollEnabled = false    scrollView.clipsToBounds = true    scrollView.contentSize = @image.size    scrollView.minimumZoomScale = 1.0    scrollView.maximumZoomScale = 4.0    scrollView.zoomScale = 0.3    scrollView.delegate = self    imageView = UIImageView.alloc.initWithImage(@image)    imageView.contentMode = UIViewContentModeScaleToFill    imageView.userInteractionEnabled = true    imageView.frame = scrollView.bounds    init  endend

ScrollView delegate

We must set a delegate for our scroll view to support zooming. The delegateobject must conform to the UIScrollViewDelegate protocol. This is the reasonwe are setting scrollView.delegate = self above. The delegate class mustimplement viewForZoomingInScrollView and scrollViewDidZoom methods.

def viewForZoomingInScrollView(scrollView)  scrollView.subviews.firstenddef scrollViewDidZoom(scrollView)  if scrollView.zoomScale != 1.0    scrollView.scrollEnabled = true  else    scrollView.scrollEnabled = false  endend

These two methods added above allow the scrollView to support pinch to zoom.

Supporting orientation changes

There is one more thing to do if we want to support orientations changes. Weneed to add the following methods:

def shouldAutorotateToInterfaceOrientation(*)  trueenddef viewDidLayoutSubviews  @scrollView.frame = self.view.boundsend

We have to set the scrollView's frame to view bounds in viewDidLayoutSubviewsso that the scrollView frame is resized when the device orientation changes.

That's it. With all those changes now our app supports orientation change andnow we are able to pinch and zoom images.

UIImage and UIImageOrientation

UIImagein iOS has a property called UIImageOrientation. Image orientation affects theway the image data is displayed when drawn. The api docs mention that bydefault, images are displayed in the up orientation. However, if the image hasassociated metadata (such as EXIF information), then this property contains theorientation indicated by that metadata.

After usingUIImagePickerControllerto take an image using the iPhone camera, I was using BubbleWrap to send theimage to a webserver. When the image is taken in landscape/portrait mode, thenthe image appeared fine when it is viewed in the browser. But, when the image issent back via api and is shown on the iphone, the image is rotated by 90 degreesif the image is taken in portrait mode. In exif metadata, iOS incorrectly setsthe orientation to UIImageOrientationRight .

Here is how I fixed the image orientation issue:

if image.imageOrientation == UIImageOrientationUp  return_image = imageelse  UIGraphicsBeginImageContextWithOptions(image.size, false, image.scale)  image.drawInRect([[0,0], image.size])  normalized_image = UIImage.UIGraphicsGetImageFromCurrentImageContext  UIGraphicsEndImageContext()  return_image = normalized_imageend

First, we are checking the image orientation of the image we have in hand. Ifthe image orientation is UIImageOrientationUp, we don't have to change anything.Otherwise we are redrawing the image and returning the normalized image.

class LoginViewController < Formotion::FormController  def viewDidLoad    super    view = UIView.alloc.init    view.backgroundColor = 0x838E61.uicolor    self.tableView.backgroundView = view  endend

After the login view is done loading, I'm creating a new UIView and setting itsbackground color. Then this UIView object is set as the background view toformotion's table view.

Setting header image

If you want to add some branding to the login form, you can add a image to theform's header by adding the below code to viewDidLoad:

header_image = UIImage.imageNamed('header_image_name.png')header_view = UIImageView.alloc.initWithImage(header_image)self.tableView.tableHeaderView = header_view

We are creating a UIImageView and initializing it with the image we want toshow in the header. Now, set the tableview's tableHeaderView value to theUIImageView we created.

irb(main):001:0> 200 * (7.0/100)=> 14.000000000000002

7 % of 200 should be 14. But float is returning 14.000000000000002 .

In order to ensure that calculation is right make sure that all the actorsparticipating in calculation is of classBigDecimal. Here is how same operation can be performed using BigDecimal .

irb(main):003:0> result = BigDecimal.new(200) * ( BigDecimal.new(7)/BigDecimal.new(100))=> #<BigDecimal:7fa5eefa1720,'0.14E2',9(36)>irb(main):004:0> result.to_s=> "14.0"

As we can see BigDecimal brings much more accurate result.

Converting money to cents

In order to charge the credit card using Stripe we neededto have the amount to be charged in cents. One way to convert the value in centswould be

amount  = BigDecimal.new(200) * ( BigDecimal.new(7)/BigDecimal.new(100))puts (amount * 100).to_i #=> 1400

Above method works but I like to delegate the functionality of making money outof a complex BigDecimal value to gem likemoney . In this project we are usingactivemerchant which depends onmoney gem . So we get money gem for free. You might have to add money gem toGemfile if you want to use following technique.

money gem lets you get a money instance out of BigDecimal.

amount  = BigDecimal.new(200) * ( BigDecimal.new(7)/BigDecimal.new(100))amount_in_money = amount.to_moneyputs amount_in_money.cents #=> 1400

Stay in BigDecimal or money mode for calculation

If you are doing any sort of calculation then all participating elements must beeither BigDecimal or Money instance. It is best if all the elements are of thesame type.

backtick

1. Returns standard output

backtick returns thestandard output(stdout) of the operation.

output = `pwd`puts "output is #{output}"

$ ruby main.rboutput is /Users/neerajsingh/code/misc

backtick does not capture STDERR . If you want to learn about STDERR thencheckout thisexcellent article.

You can redirect STDERR to STDOUT if you want to capture STDERR usingbacktick.

output = `grep hosts /private/etc/* 2>&1`

2. Exception is passed on to the main program

Backtick operation forks the master process and the operation is executed in anew process. If there is an exception in the sub-process then that exception isgiven to the main process and the main process might terminate if exception isnot handled.

In the following case I am executing xxxxx which is not a valid executablename.

output = `xxxxxxx`puts "output is #{output}"

Result of above code is given below. Notice that puts was never executedbecause the backtick operation raised exception.

$ ruby main.rbmain.rb:1:in ``': No such file or directory - xxxxxxx (Errno::ENOENT)from main.rb:1:in `<main>'

3. Blocking operation

Backtick is a blocking operation. The main application waits until the result ofbacktick operation completes.

4. Checking the status of the operation

To check the status of the backtick operation you can execute $?.success?

output = `ls`puts "output is #{output}"puts $?.success?

Notice that the last line of the result contains true because the backtickoperation was a success.

$ ruby main.rboutput is lab.rbmain.rbtrue

5. String interpolation is allowed within the ticks

cmd = 'ls'`#{cmd}`

6. Different delimiter and string interpolation

%x does the same thing as backtick. It allows you to have different delimiter.

output = %x[ ls ]output = %x{ ls }

backtick runs the command in subshell. So shell features like stringinterpolation and wild card can be used. Here is an example.

$ irb> dir = '/etc'> %x<ls -al #{dir}>=> "lrwxr-xr-x@ 1 root  wheel  11 Jan  5 21:10 /etc -> private/etc"

If you are building a script which you mean to run on your laptops and not onserver then most likely if there is an exception then you want the script toabort. For such cases backtick is the best choice.

For example let's say that I want to write a script to make my repo up-to-dateautomatically. The command would be something like this.

cd directory_name && git checkout main && git pull origin main

If there is any error while executing this command then you want to have thefull access to the exception so that you could debug. In such cases the best wayto execute this command is as shown below.

cmd = "cd #{directory_name} && git checkout main && git pull origin main"%x[ cmd ]

system

The system command runs ina subshell.

Just like backtick, system is a blocking operation.

Since system command runs in a subshell it eats up all the exceptions. So themain operation never needs to worry about capturing an exception raised from thechild process.

output = system('xxxxxxx')puts "output is #{output}"

Result of the above operation is given below. Notice that even when exception israised the main program completes and the output is printed. The value of outputis nil because the child process raised an exception.

$ ruby main.rboutput is

system returns true if the command was successfully performed ( exit statuszero ) . It returns false for non zero exit status. It returns nil ifcommand execution fails.

system("command that does not exist")  #=> nilsystem("ls")                           #=> truesystem("ls | grep foo")                #=> false

system sets the global variable $? to the exit status of the process. Rememberthat a value of zero means the operation was a success.

The biggest issue with system command is that it's not possible to capture theoutput of the operation.

exec

Kernel#exec replacesthe current process by running the external command.

Let's see an example. Here I am in irb and I am going to execute exec('ls').

$ irbe1.9.3-p194 :001 > exec('ls')lab.rb  main.rbnsingh ~/neerajsingh$

I see the result but since the irb process was replaced by the exec process Iam no longer in irb .

Behind the scene both system and backtick operations use fork to fork thecurrent process and then they execute the given operation using exec .

Since exec replaces the current process it does not return anything. It printsthe output on the screen. There is no way to know if the operation was a"success" or a "failure" and hence it's not recommended to use exec.

sh

sh actually calls systemunder the hood. However it is worth a mention here. This method is added byFileUtils in rake. It allows an easy way to check the exit status of thecommand.

require 'rake'sh %w(xxxxx) do |ok, res|   if !ok     abort 'the operation failed'   endend

popen3

If you are going to capture stdout and stderr then you should usepopen3since this method allows you to interact with stdin, stdout and stderr .

I want to execute git push heroku master programmatically and I want tocapture the output. Here is my code.

require 'open3'cmd = 'git push heroku master'Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|  puts "stdout is:" + stdout.read  puts "stderr is:" + stderr.readend

And here is the output. It has been truncated since rest of output is notrelevant to this discussion.

stdout is:stderr is:-----> Heroku receiving push-----> Ruby/Rails app detected-----> Installing dependencies using Bundler version 1.2.1

The important thing to note here is that when I execute the programruby lab.rb I do not see any output on my terminal for first 10 seconds. ThenI see the whole output as one single dump.

The other thing to note is that heroku is writing all this output to stderrand not to stdout .

Above solution works but it has one major drawback. The push to heroku mighttake 10 to 20 seconds and for this period we do not get any feedback on theterminal. In reality when we execute git push heroku master we start seeingresult on our terminal one by one as heroku is processing things.

So we should capture the output from heroku as it is being streamed rather thandumping the whole output as one single chunk of string at the end of processing.

Here is the modified code.

require 'open3'cmd = 'git push heroku master'Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|  while line = stderr.gets    puts line  endend

Now when I execute above command using ruby lab.rb I get the output on myterminal incrementally as if I had typed git push heroku master .

Here is another example of capturing streaming output.

require 'open3'cmd = 'ping www.google.com'Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|  while line = stdout.gets    puts line  endend

In the above case you will get the output of ping on your terminal as if you hadtyped ping www.google.com on your terminal .

Now let's see how to check if command succeeded or not.

require 'open3'cmd = 'ping www.google.com'Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|  exit_status = wait_thr.value  unless exit_status.success?    abort "FAILED !!! #{cmd}"  endend

popen2e

popen2eis similar to popen3 but merges the standard output and standard error .

require 'open3'cmd = 'ping www.google.com'Open3.popen2e(cmd) do |stdin, stdout_err, wait_thr|  while line = stdout_err.gets    puts line  end  exit_status = wait_thr.value  unless exit_status.success?    abort "FAILED !!! #{cmd}"  endend

In all other areas this method works similar to popen3 .

Process.spawn

Kernel.spawnexecutes the given command in a subshell. It returns immediately with theprocess id.

irb(main)> pid = Process.spawn("ls -al")=> 81001

Class is meant for both data and behavior

Lets look at this ruby code.

class Util  def self.double(i)   i*2  endendUtil.double(4) #=> 8

Here we have a Util class. But notice that all the methods on this class areclass methods. This class does not have any instance variables. Usually a classis used to carry both data and behavior and ,in this case, the Util class hasonly behavior and no data.

Similar utility tools in ruby

Now to get some perspective on this discussion lets look at some ruby methodsthat do similar thing. Here are a few.

require 'base64'Base64.encode64('hello world') #=> "aGVsbG8gd29ybGQ=\n"require 'benchmark'Benchmark.measure { 10*2000 }require 'fileutils'FileUtils.chmod 0644, 'test.rb'Math.sqrt(4) #=> 2

In all the above cases the class method is invoked without creating an instancefirst. So this is similar to the way I used Util.double .

However lets see what is the class of all these objects.

Base64.class #=> ModuleBenchmark.class #=> ModuleFileUtils.class #=> ModuleMath.class #=> Module

So these are not classes but modules. That begs the question why the smart guysat ruby-core implemented them as modules instead of creating a class the way Idid for Util.

Reason is that Class is too heavy for creating only methods like double. As wediscussed earlier a class is supposed to have both data and behavior. If theonly thing you care about is behavior then ruby suggests to implement it as amodule.

extend self is the answer

Before I go on to discuss extend self here is how my Util class will lookafter moving from Class to Module.

module Util  extend self  def double(i)    i * 2  endendputs Util.double(4) #=> 8

So how does extend self work

First lets see what extend does.

module M def double(i)  i * 2 endendclass Calculator  extend Mendputs Calculator.double(4)

In the above case Calculator is extending module M and hence all theinstance methods of module M are directly available to Calculator.

In this case Calculator is a class that extended the module M. HoweverCalculator does not have to be a class to extend a module.

Now lets try a variation where Calculator is a module.

module M def double(i)  i * 2 endendmodule Calculator  extend Mendputs Calculator.double(4) #=> 8

Here Calculator is a module that is extending another module.

Now that we understand that a module can extend another module look at the abovecode and question why module M is even needed. Why can't we move the methoddouble to module Calculator directly. Let's try that.

module Calculator  extend Calculator   def double(i)    i * 2   endendputs Calculator.double(4) #=> 8

I got rid of module M and moved the method double inside moduleCalculator. Since module M is gone I changed from extend M toextend Calculator.

One last fix.

Inside the module Calculator what is self. self is the module Calculatoritself. So there is no need to repeat Calculator twice. Here is the finalversion

module Calculator  extend self   def double(i)    i * 2   endendputs Calculator.double(4) #=> 8

Converting A Class into a Module

Every time I would encounter code like extend self my brain will pause for amoment. Then I would google for it. Will read about it. Three months later Iwill repeat the whole process.

The best way to learn it is to use it. So I started looking for a case to useextend self. It is not a good practice to go hunting for code to apply an ideayou have in your mind but here I was trying to learn.

Here is a before snapshot of methods from Util class I used in a project.

class Util  def self.config2hash(file); end  def self.in_cents(amount); end  def self.localhost2public_url(url, protocol); endend

After using extend self code became

module Util  extend self  def config2hash(file); end  def in_cents(amount); end  def localhost2public_url(url, protocol); endend

Much better. It makes the intent clear and ,I believe, it is in line with theway ruby would expect us to use.

Another usage inline with how Rails uses extend self

Here I am building an ecommerce application and each new order needs to get anew order number from a third party sales application. The code might look likethis. I have omitted the implementation of the methods because they are notrelevant to this discussion.

class Order  def amount; end  def buyer; end  def shipped_at; end  def number    @number || self.class.next_order_number  end  def self.next_order_number; 'A100'; endendputs Order.new.number #=> A100

Here the method next_order_number might be making a complicated call toanother sales system. Ideally the class Order should not expose methodnext_order_number . So we can make this method private but that does notsolve the root problem. The problem is that model Order should not know howthe new order number is generated. Well we can move the methodnext_order_number to another Util class but that would create too muchdistance.

Here is a solution using extend self.

module Checkout  extend self  def next_order_number; 'A100'; end  class Order    def amount; end    def buyer; end    def shipped_at; end    def number      @number || Checkout.next_order_number    end  endendputs Checkout::Order.new.number #=> A100

Much better. The class Order is not exposing method next_order_number and thismethod is right there in the same file. No need to open the Util class.

To see practical examples of extend self please look at Rails source code andsearch for extend self. You will find some interesting usage.

This is my first serious attempt to learn usage of extend self so that nexttime when I come across such code my brain does not freeze. If you think I havemissed out something then do let me know.

All objects have to_s method

to_s method is define in Object class and hence all ruby objects have methodto_s.

Certain methods always call to_s method. For example when we do stringinterpolation then to_s method is called. puts invokes to_s method too.

class Lab def to_s  'to_s' end def to_str  'to_str' endendl = Lab.newputs "#{l}" #=> to_sputs l #=> to_s

to_s is simply the string representation of the object.

Before we look at to_str let's see a case where ruby raises error.

e = Exception.new('not sufficient fund')# case 1puts e# case 2puts "notice: #{e}"# case 3puts "Notice: " + e

Here is the result

not sufficient fundNotice: not sufficient fund`+': can't convert Exception into String (TypeError)

In the first two cases the to_s method of object e was printed.

However in case '3' ruby raised an error.

Let's read the error message again.

`+': can't convert Exception into String (TypeError)

In this case on the left hand side we have a string object. To this stringobject we are trying to add object e. Ruby could have called to_s method one and could have produced the result. But ruby refused to do so.

Ruby refused to do so because it found that the object we are trying to add tostring is not of type String. When we call to_s we get the stringrepresentation of the string. But the object might or might not be behaving likea string.

Here we are not looking for the string representation of e. What we want isfor e to behave a like string. And that is where to_str comes in picture. Ihave a few more examples to clear this thing so hang in there.

What is to_str

If an object implements to_str method then it is telling the world that myclass might not be String but for all practical purposes treat me like astring.

So if we want to make exception object behave like a string then we can addto_str method to it like this.

e = Exception.new('not sufficient fund')def e.to_str  to_sendputs "Notice: " + e #=> Notice: not sufficient fund

Now when we run the code we do not get any exception.

What would happen if Fixnum has to_str method

Here is an example where ruby raises exception.

i = 10puts '7' + i #=> can't convert Fixnum into String (TypeError)

Here Ruby is saying that Fixnum is not like a string and it should not be addedto String.

We can make Fixnum to behave like a string by adding a to_str method.

class Fixnum  def to_str    to_s  endendi = 10puts '7' + i #=> 710

The practical usage of this example can be seen here.

irb(main):002:0> ["hello", "world"].join(1)TypeError: no implicit conversion of Fixnum into String

In the above case ruby is refusing to invoke to_s on "1" because it knows thatadding "1" to a string does not feel right.

However we can add method to_str to Fixnum as shown in the last section andthen we will not get any error. In this case the result will be as shown below.

irb(main):008:0> ["hello", "world"].join(1)=> "hello1world"

A real practical example of defining to_str

I tweeted abouta quick lesson in to_s vs to_strand a few people asked me to expand on that. Lets see what is happening here.

Before the refactoring was done Path is a subclass of String. So it isString and it has all the methods of a string.

As part of refactoring Path is no longer extending from String. However forall practical purposes it acts like a string. This line is important and I amgoing to repeat it. For all practical purposes Path here is like a String.

Here we are not talking about the string representation of Path. Here Pathis so close to String that practically it can be replaced for a string.

So in order to be like a String class Path should have to_str method andthat's exactly what was done as part of refactoring.

During discussion with my friends someone suggested instead of defining to_strtenderlove could have just defined to_s and the result would have been same.

Yes the result would be same whether you have defined to_s or to_str if youdoing puts.

puts Path.new('world')

However in the following case just defining to_s will cause error. Only byhaving to_str following case will work.

puts 'hello ' + Path.new('world')

So the difference between defining to_s and to_str is not just what you seein the output.

Conclusion

If a class defines to_str then that class is telling the world that althoughmy class is not String you can treat me like a String.

Usage of alias

class User  def full_name    puts "Johnnie Walker"  end  alias name full_nameendUser.new.name #=>Johnnie Walker

Usage of alias_method

class User  def full_name    puts "Johnnie Walker"  end  alias_method :name, :full_nameendUser.new.name #=>Johnnie Walker

First difference you will notice is that in case of alias_method we need touse a comma between the "new method name" and "old method name".

alias_method takes both symbols and strings as input. Following code wouldalso work.

alias_method 'name', 'full_name'

That was easy. Now let's take a look at how scoping impacts usage of alias andalias_method .

Scoping with alias

class User  def full_name    puts "Johnnie Walker"  end  def self.add_rename    alias_method :name, :full_name  endendclass Developer < User  def full_name    puts "Geeky geek"  end  add_renameendDeveloper.new.name #=> 'Gekky geek'

In the above case method "name" picks the method "full_name" defined in"Developer" class. Now let's try with alias.

class User  def full_name    puts "Johnnie Walker"  end  def self.add_rename    alias :name :full_name  endendclass Developer < User  def full_name    puts "Geeky geek"  end  add_renameendDeveloper.new.name #=> 'Johnnie Walker'

With the usage of alias the method "name" is not able to pick the method"full_name" defined in Developer.

This is because alias is a keyword and it is lexically scoped. It means ittreats self as the value of self at the time the source code was read . Incontrast alias_method treats self as the value determined at the run time.

Overall my recommendation would be to use alias_method. Since alias_methodis a method defined in class Module it can be overridden later and it offersmore flexibility.

Here is an example.

> 'A'.unpack('b*')=> ["10000010"]

In the above case 'A' is a string which is being stored and using unpack I amtrying to read the bit value. The ASCII table saysthat ASCII value of 'A' is 65 and the binary representation of 65 is 10000010.

Here is another example.

> 'A'.unpack('B*')=> ["01000001"]

Notice the difference in result from the first case. What's the differencebetween b* and B*. In order to understand the difference first lets discussMSB and LSB.

Most significant bit vs Least significant bit

All bits are not created equal. C has ascii value of 67. The binary value of67 is 1000011.

First let's discuss MSB (most significant bit) style . If you are following MSBstyle then going from left to right (and you always go from left to right) thenthe most significant bit will come first. Because the most significant bit comesfirst we can pad an additional zero to the left to make the number of bitseight. After adding an additional zero to the left the binary value looks like01000011.

If we want to convert this value in the LSB (Least Significant Bit) style thenwe need to store the least significant bit first going from left to right. Givenbelow is how the bits will be moved if we are converting from MSB to LSB. Notethat in the below case position 1 is being referred to the leftmost bit.

move value 1 from position 8 of MSB to position 1 of LSBmove value 1 from position 7 of MSB to position 2 of LSBmove value 0 from position 6 of MSB to position 3 of LSBand so on and so forth

After the exercise is over the value will look like 11000010.

We did this exercise manually to understand the difference betweenmost significant bit and least significant bit. However unpack method candirectly give the result in both MSB and LSB. The unpack method can take bothb* and B* as the input. As per the ruby documentation here is thedifference.

B | bit string (MSB first)b | bit string (LSB first)

Now let's take a look at two examples.

> 'C'.unpack('b*')=> ["11000010"]> 'C'.unpack('B*')=> ["01000011"]

Both b* and B* are looking at the same underlying data. It's just that theyrepresent the data differently.

Different ways of getting the same data

Let's say that I want binary value for string hello . Based on the discussionin the last section that should be easy now.

> "hello".unpack('B*')=> ["0110100001100101011011000110110001101111"]

The same information can also be derived as

> "hello".unpack('C*').map {|e| e.to_s 2}=> ["1101000", "1100101", "1101100", "1101100", "1101111"]

Let's break down the previous statement in small steps.

> "hello".unpack('C*')=> [104, 101, 108, 108, 111]

Directive C* gives the 8-bit unsigned integer value of the character. Notethat ascii value of h is 104, ascii value of e is 101 and so on.

Using the technique discussed above I can find hex value of the string.

> "hello".unpack('C*').map {|e| e.to_s 16}=> ["68", "65", "6c", "6c", "6f"]

Hex value can also be achieved directly.

> "hello".unpack('H*')=> ["68656c6c6f"]

High nibble first vs Low nibble first

Notice the difference in the below two cases.

> "hello".unpack('H*')=> ["68656c6c6f"]> "hello".unpack('h*')=> ["8656c6c6f6"]

As per ruby documentation for unpack

H | hex string (high nibble first) h | hex string (low nibble first)

A byte consists of 8 bits. A nibble consists of 4 bits. So a byte has twonibbles. The ascii value of 'h' is 104. Hex value of 104 is 68. This 68 isstored in two nibbles. First nibble, meaning 4 bits, contain the value 6 andthe second nibble contains the value 8. In general we deal with high nibblefirst and going from left to right we pick the value 6 and then 8.

However if you are dealing with low nibble first then low nibble value 8 willtake the first slot and then 6 will come. Hence the result in "low nibblefirst" mode will be 86.

This pattern is repeated for each byte. And because of that a hex value of68 65 6c 6c 6f looks like 86 56 c6 c6 f6 in low nibble first format.

Mix and match directives

In all the previous examples I used *. And a * means to keep going as longas it has to keep going. Lets see a few examples.

A single C will get a single byte.

> "hello".unpack('C')=> [104]

You can add more Cs if you like.

> "hello".unpack('CC')=> [104, 101]> "hello".unpack('CCC')=> [104, 101, 108]> "hello".unpack('CCCCC')=> [104, 101, 108, 108, 111]

Rather than repeating all those directives, I can put a number to denote howmany times you want previous directive to be repeated.

> "hello".unpack('C5')=> [104, 101, 108, 108, 111]

I can use * to capture al the remaining bytes.

> "hello".unpack('C*')=> [104, 101, 108, 108, 111]

Below is an example where MSB and LSB are being mixed.

> "aa".unpack('b8B8')=> ["10000110", "01100001"]

pack is reverse of unpack

Method pack is used to read the stored data. Let's discuss a few examples.

>  [1000001].pack('C')=> "A"

In the above case the binary value is being interpreted as8 bit unsigned integer and the result is 'A'.

> ['A'].pack('H')=> "\xA0"

In the above case the input 'A' is not ASCII 'A' but the hex 'A'. Why is it hex'A'. It is hex 'A' because the directive 'H' is telling pack to treat inputvalue as hex value. Since 'H' is high nibble first and since the input has onlyone nibble then that means the second nibble is zero. So the input changes from['A'] to ['A0'] .

Since hex value A0 does not translate into anything in the ASCII table thefinal output is left as it and hence the result is \xA0. The leading \xindicates that the value is hex value.

Notice the in hex notation A is same as a. So in the above example I canreplace A with a and the result should not change. Let's try that.

> ['a'].pack('H')=> "\xA0"

Let's discuss another example.

> ['a'].pack('h')=> "\n"

In the above example notice the change. I changed directive from H to h.Since h means low nibble first and since the input has only one nibble thevalue of low nibble becomes zero and the input value is treated as high nibblevalue. That means value changes from ['a'] to ['0a']. And the output will be\x0A. If you look at ASCII table then hex value A is ASCII value 10 which isNL line feed, new line. Hence we see \n as the output because it represents"new line feed".

Usage of unpack in Rails source code

I did a quick grep in Rails source code and found following usage of unpack.

email_address_obfuscated.unpack('C*')'mailto:'.unpack('C*')email_address.unpack('C*')char.unpack('H2')column.class.string_to_binary(value).unpack("H*")data.unpack("m")s.unpack("U\*")

Already we have seen the usage of directive C* and H for unpack. Thedirective m gives the base64 encoded value and the directive U* gives theUTF-8 character. Here is an example.

> "Hello".unpack('U*')=> [72, 101, 108, 108, 111]

Testing environment

Above code was tested with ruby 1.9.2 .

French version of this article is availablehere .

Default value of Hash

If I want a hash to have a default value then that's easy.

h = Hash.new(0)puts h['usa'] #=> 0

Above code will give me a fixed value if key is not found. If I want dynamicvalue then I can use block form.

h = Hash.new{|h,k| h[k] = k.upcase}puts h['usa'] #=> USAputs h['india'] #=> INDIA

Default value is hash

If I want the default value to be a hash then it seems easy but it falls apartsoon.

h = Hash.new{|h,k| h[k] = {} }puts h['usa'].inspect #=> {}puts h['usa']['ny'].inspect #=> nilputs h['usa']['ny']['nyc'].inspect #=> NoMethodError: undefined method `[]' for nil:NilClass

In the above if a key is missing for h then it returns a hash. However thatreturned hash is an ordinary hash which does not have a capability of returninganother hash if a key is missing.

This is where default_proc comes into picture.hash.default_procreturns the block which was passed to Hash.new .

h = Hash.new{|h,k| Hash.new(&h.default_proc)}puts h['usa']['ny']['nyc'].inspect #=> {}

Following code will print 99 as the output.

class Klass  def initialize    @secret = 99  endendputs Klass.new.instance_eval { @secret }

Nothing great there. However try passing a parameter to instance_eval .

puts Klass.new.instance_eval(self) { @secret }

You will get following error.

wrong number of arguments (1 for 0)

So instance_eval does not allow you to pass parameters to a block.

How to get around to the restriction that instance_eval does not accept parameters

instance_exec was added to ruby 1.9 and it allows you to pass parameters to aproc. This feature has been backported to ruby 1.8.7 so we don't really needruby 1.9 to test this feature. Try this.

class Klass  def initialize    @secret = 99  endendputs Klass.new.instance_exec('secret') { |t| eval"@#{t}" }

Above code works. So now we can pass parameters to block. Good.

Changing value of self

Another feature of instance_exec is that it changes the value of self. Toillustrate that I need to give a longer example.

module Kernel  def singleton_class    class << self      self    end  endendclass Human  proc = lambda { puts 'proc says my class is ' + self.name.to_s }  singleton_class.instance_eval do    define_method(:lab)  do      proc.call    end  endendclass Developer < HumanendHuman.lab # class is HumanDeveloper.lab # class is Human ; oops

Notice that in that above case Developer.lab says "Human". And that is theright answer from ruby perspective. However that is not what I intended. rubystores the binding of the proc in the context it was created and hence itrightly reports that self is "Human" even though it is being called byDeveloper.

Go tohttp://facets.rubyforge.org/apidoc/api/core/index.htmland look for instance_exec method. The doc says

<blockquote>Evaluate the block with the given arguments within the context of this object,so self is set to the method receiver.</blockquote>

It means that instance_exec evaluates self in a new context. Now try the samecode with instance_exec .

module Kernel  def singleton_class    class << self      self    end  endendclass Human  proc = lambda { puts 'proc says my class is ' + self.name.to_s }  singleton_class.instance_eval do    define_method(:lab)  do      self.instance_exec &proc    end  endendclass Developer < HumanendHuman.lab # class is HumanDeveloper.lab # class is Developer

In this case Developer.lab says Developer and not Human.

You can also checkout this page (Link is not available) which has much moredetailed explanation of instance_exec and also emphasizes that instance_execdoes pass a new value of self .

instance_exec is so useful that ActiveSupport needs it. And since ruby 1.8.6does not have it ActiveSupport has code to support it.

I came across instance_exec issue while resolving#4507 rails ticket. The final solution did not need instance_exec but I learned a bit about it.

In order to reload the class, Rails first has to unload . That unloading isdone something like this.

# unload User classObjet.send(:remove_const, :User)

However a class might have other constants and they need to be unloaded too.Before you unload those constants you need to know all the constants that aredefined in the class that is being loaded. Long story short rails keep track ofevery single constant that is loaded when it loads User or UserController.

Dependency mechanism is not perfect

Sometimes dependency mechanism by rails lets a few things fall through thecrack. Try following case.

require 'open-uri'class UsersController < ApplicationController  def index    open("http://www.ruby-lang.org/") {|f| }    render :text => 'hello'  endend

Start the server in development mode and visit http://localhost:3000/users .First time every thing will come up fine. Now refresh the page. This time youshould get an exception uninitialized constant OpenURI .

So what's going on.

After the page is served the very first time then at the end of response railswill unload all the constants that were autoloaded including UsersController.However while unloading UsersContorller rails will also unload OpenURI.

When the page is refreshed then UsersController will be loaded andrequire 'open-uri' will be called. However that require will return false.

Why require returns false

Try the following test case in irb.

step 1

irb(main):002:0> require 'ostruct'=> true

step 2

irb(main):005:0* Object.send(:remove_const, :OpenStruct)=> OpenStruct

step 3 : ensure that OpenStruct is truly removed

irb(main):006:0> Object.send(:remove_const, :OpenStruct)NameError: constant Object::OpenStruct not defined        from (irb):6:in `remove_const'        from (irb):6:in `send'        from (irb):6

step 4

irb(main):007:0> require 'ostruct'=> false

step 5

irb(main):009:0> OpenStruct.newNameError: uninitialized constant OpenStruct        from (irb):9

Notice that in the above case in step 4 require returns false. 'require'checks against $LOADED_FEATURES. When OpenStruct was removed then it was notremoved from $LOADED_FEATURES and hence ruby thought ostruct is alreadyloaded.

How to get around to this issue.

require loads only once. However load loads every single time. In stead of'require', 'load' could be used in this case.

irb(main):001:0> load 'ostruct.rb'=> trueirb(main):002:0> OpenStruct.new=> #<OpenStruct>

Back to the original problem

In our rails application refresh of the page is failing. To get around to thatissue use require_dependency instead of require. require_dependency is arails thing. Under the hood rails does the same trick we did in the previousstep. Rails calls kernel.load to load the constants that would fail if requirewere used.

5.times do  putc('.')  sleep(2)end

I was hoping that I will get a dot after every two seconds. But that's not whathappens when you run the code. I see nothing for first 10 seconds then I seefive dots in one shot. This is not what I wanted.

I started looking around at the documentation for IO class and found the methodcalled sync= which ifset to true will flush all output immediately to the underlying OS.

The reason why OS does not flush immediately is to minimize the IO operationwhich is usually slow. However in this case you are asking OS to not to bufferand to flush the output immediately.

$stdout.sync = true5.times do  putc('.')  sleep(2)end

While developing rails application you have must seen this

Called id for nil, which would mistakenly be 4  if you reallywanted the id of nil, use object_id

We all know that this message is added by Rails and it is called whiny nil .If you open your config/development.rb file you will see

# Log error messages when you accidentally call methods on nil.config.whiny_nils = true

Simply stated it means that if the application happens to invoke id on a nilobject then throw an error. Rails assumes that under no circumstance a developerwants to find id of a nil object. So this must be an error case and Rails throwsan exception.

The question I have is why 4. Why Matz chose the id of nil to be 4.This awesome presentationon 'Ruby Internals' has the answer.

In short Matz decided to have all the odd numbers reserved for numerical values.Check this out.

> irb>> 0.id(irb):1: warning: Object#id will be deprecated; use Object#object_id=> 1>> 1.id(irb):2: warning: Object#id will be deprecated; use Object#object_id=> 3>> 2.id(irb):3: warning: Object#id will be deprecated; use Object#object_id=> 5>> 3.id(irb):4: warning: Object#id will be deprecated; use Object#object_id=> 7

Id 1,3,5 and 7 are taken by 0,1,2 and 3.

Now we are left with the id 0,2,4 and higher values.

> irb> FALSE.id(irb):5: warning: Object#id will be deprecated; use Object#object_id=> 0>> TRUE.id(irb):6: warning: Object#id will be deprecated; use Object#object_id=> 2

FALSE had the id 0 and TRUE has the id 2.

Now the next available id left is 4 and that is taken by NIL.

> irb>> NIL.id(irb):7: warning: Object#id will be deprecated; use Object#object_id=> 4

We won't even be discussing this issue once 1.9 comes out where we will have touse object_id and then this won't be an issue.

You can follow more discussion about this article atHacker news .