How to Install and Run Elasticsearch as a Windows Service

Planning to keep this as simple and helpful as I can.
This whole process can be done in just a few minutes, assuming your downloads happen quickly.

Prerequisites:

  1. Log in to your server with credentials that allow software installation.
  2. Download Elasticsearch (latest. 1.5.2 used in this example) as a ZIP file from https://www.elastic.co/downloads/elasticsearch
  3. Unzip the file you just downloaded.
  4. Copy the child Elasticsearch1.5.2 folder and contents to C:\elasticsearch-1.5.2, or wherever you want it.
    • (You should have this path: D:\elasticsearch-1.5.2\bin )
  5. Download and install Java JDK 64 bit version from: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
  6. Set JAVA_HOME environment variable:
  7. java_home

  8. Install Elasticsearch as a service:
  9. (Here’s the original reference if needed)

    1. Run command prompt as administrator
    2. Navigate to C:\elasticsearch-1.5.2\bin
    3. Type service.bat install and hit Enter.
    4. It should install as a 64 bit service:
    5. es_console

    6. Close command prompt
    7. Open services console, and find Elasticsearch service.
    8. Open Elasticsearch service properties and set the service to start automatically and then start service.
    9. In a browser on the server, go to http://localhost:9200/
    10. You should see something like this (Note that the “name” is randomly selected from a list of around 3000 Marvel characters.):
    11. es_screenprint

  10. If you see that, you have successfully installed Elasticsearch in a cluster with one node, and you rock.
  11. Setting up multiple nodes is beyond the scope of this article.

    Elasticsearch is amazing, and running as a Windows Service is nice, because you don’t have to make sure you start it manually each time.

Amazing Speed: Elasticsearch for the .NET Developer. Codestock, 2015

This presentation is from a talk I gave at Codestock, 2015, which had over 900 attendees. There were between 50 and 60 people who attended this talk.

Here’s the text of the slides:
Amazing Speed:
Elasticsearch for the .NET Developer
4:05 PM / 301-D
Adrian Carr
With special guest- Shaunak Kashyap, Developer Advocate, Elastic.co
A little about me…
• Software Engineer & Team Lead
o The White Stone Group in Knoxville, TN.
• Previously
o Jewelry Television
o Alltel/Fidelity
o One terrible non-profit in Boone, NC that I don’t like to talk about.
• Several small startups

Experience:
• Full stack developer, C#, Ruby/Rails, JavaScript, Java, etc……. “Business Problem Solver”

A little background…
• We have some large clients, with lots of patients.

Really slow sometimes…
• Some legacy database decisions that weren’t so good back in the day.
• Worked just fine when volume was lower.
• Re-architecting would mean changing too many apps.
• Maybe you just have too much data to search efficiently, or it isn’t structured for search.

I’ve seen this before…

• Jewelry Television
• At the time, a lot of the database design wasn’t very good, but it worked.

o ~$200 million when I started in 2005
o ~$550 million three years later

The JTV Solution:

• It was awesome!

I’m a Business Problem Solver…

So….

A little research…
I found Elasticsearch -built on top of Lucene.

I found:
Wikipedia

And here:
GitHub

And here:
StackOverflow

What is Elasticsearch?
Elasticsearch is a distributed, open source search and analytics engine, designed for horizontal scalability, reliability, and easy management

So, I did a proof of concept

Terminology
• Node: DB Instance. (Java process running Elasticsearch)
• Cluster: Database Cluster- One or more nodes with same cluster name
• Index: Database, logical grouping of tables
• Type: Database Table
• Document: Like a row. JSON, key-value pairs
• Fields: Columns
• Shard: Worker processes. (These mostly happen automagically by Elastic)

Tools
• Elasticsearch (Obviously)
o Demo of how to install and run. (It’s easy.)

• Browser http://localhost:9200/, etc.

• Sense- a fantastic Chrome plugin

• C# with Nest

How to Load & Synchronize Data?
• It depends…

This is what I did:
• SQL Server Triggers on data that matters
• Windows Service
o Nest, C#, Elastic’s Bulk API (Found 10,000 rows at a time to be the sweet spot for insertion speed)

Gotchas
• Elasticsearch.net vs Nest
• Documentation on Elasticsearch is fantastic. Documentation on Nest is very sparse.

More Gotchas
• It’s a different way of thinking.
o Not RDBMS
o Joins? Nope
• Case sensitivity
• Index size- disk space can grow a lot
• Integration with existing application. Are your existing users going to have their cheese moved?

There is much more to know…

• Analyzers- These can be very complex
• Scaling- Elastic has a lot built in, but it can be tuned quite a bit.
• Field weighting- Relevance of blog post vs body vs comments
• This is just a start.

Questions?
• Code samples will eventually be at http://adriancarr.com
• Contact:

o adriancarr@gmail.com

o shaunak@elastic.co

o https://discuss.elastic.co/

Thank You!

Things your professors probably never told you, but will make you more money and help you have fun doing it.

AITP Presentation at Pellissippi State Community College- November, 2014

Great fun presenting this to around 50-60 students and faculty at PSCC. There was a good bit of commentary and discussion that’s not captured here, but this captures most of it.

Make it searchable:

Association of Information Technology Professionals
Pellissippi State Community College
November 13, 2014 Adrian Carr
Congratulations! This can be a fantastic career.
It can really suck at times too.
A lot of this is up to you:
Things your professors probably never told you, but will make you more money and help you have fun doing it.
A bit about me…
• I’m a software engineer, team lead….
– I’ve done other things…
• I also do some software consulting
• I started at a Community College- (Attended three of them)
It was hard, but worth it

I love my job
• Not every day, but almost every day.
Getting started
1.Please graduate. It opens doors, and I’ve met a lot of people who wish they had.
2.Learn skills.
3.Build your “brand”.

Create a great resume

A bad resume

Resume Tips
• 1-2 pages
• Make it clean and easy to read.
• Make it relevant – Tailor it for the position.
• No speling erors!
I’m gonna Google you…
These can help build your brand, or work against it.
• Facebook
• Twitter
• Personal blog
• Dr Brown’s student pages
• LinkedIn.com

Who is responsible for your career? • You are.
• IT folks == Professional problem solvers
• I learn something almost every day, and I usually have to teach it to myself.
Most technologies that I work with today didn’t exist when I started…
• C#
• JSON/XML
• jQuery/Angular
• Android
• Ruby/Rails
• HTML5/CSS3
• Bootstrap
• …..
Be purposeful about it
• Home projects
• Volunteer- inside and outside of work
• Consulting- This is a great way to learn new technologies. Create a website or mobile app for a small fee for your cousin’s auto body shop…
• Change jobs every few years.
• Do things outside your comfort zone
Even if you are on the right track, you’ll get run over if you just sit there. – Will Rogers
Be valuable
• Why are you being paid?
• Are you delivering value or filling a seat?
• If it was your money you were spending on your salary, would you change the way you work?
Most programmers have too many layers of abstraction between their code and their paycheck. – Michael Neel
Solve Problems
• Ask why.
• Figure out what problem people are trying to solve.
• Always look for better ways.
We cannot solve our problems with the same thinking we used when we created them. -Albert Einstein
Value
• Make your boss look good.
• Be worth so much more than you are paid that it’s no question about whether they should keep you or not.

Luck Surface Area:
Luck Surface Area is directly proportional to the degree to which you do something you’re passionate about combined with the total number of people to whom this is effectively communicated.
3. Build your “brand”.
“Soft” Skills
• Communication
• Attitude – “Whether you think you can or think you can’t, you’re right.“ -Henry Ford
• Humility goes a long way.

Be the type of person that people want to work with.
• Bring donuts or leftover cookies
• Be patient.

People may not remember exactly what you did or what you said, but they will always remember how you made them feel.
–Tony Hsieh, Zappos.com
Get involved
• AITP is a great start.
• Codestock
• Knoxville JS
• Agile Knoxville
• Write about things you learn- somewhere.
• StackOverflow.com
• Mentoring- High School, PSCC students,
http://techco.org/

A lot of this is up to you:
Things your professors probably never told you, but will make you more money and help you have fun doing it.
You can have options
If you do things right, you have options
• You can change where you work, or you can change where you work .
•”You have brains in your head. You have feet in your shoes. You can steer yourself, any direction you choose.” – Dr. Seuss
If you do really do things right…
• You can help set the course, and make the way we work better.
http://www.virgin.com/richard-branson/the-way-you-work-is-going-to-change
Questions?
Resources Job search/market research:
• http://indeed.com
Reading:
• Code Complete – Steve McConnell
• https://gettingreal.37signals.com/
• http://theleanstartup.com/
• Leading Lean Software Development – Mary and Tom Poppendieck
Contact: http://adriancarr.com

SQL Server- Check if Current Time is in Standard Work Week

Had to figure this out today, and thought I’d post it here, because I didn’t see it anywhere else out there.
There maybe better ways of doing this. If so, please let me know.

--Check to see if current time is a weekday and between 9 and 5
DECLARE @isWorkHours AS bit

SELECT @isWorkHours = COUNT(1)
WHERE
--Monday is 2 and Friday is 6.
DATEPART(dw,GETDATE()) BETWEEN 2 AND 6
AND
--This gets times between 9:00 and 4:59
DATEPART(hour, GETDATE()) BETWEEN 9 AND 16
IF @isWorkHours = 1
BEGIN
SELECT 1
END

IF @isWorkHours = 0
BEGIN
SELECT 0
END

Money, Process, and Culture- Lean and Agile Applied

This is the Powerpoint for a talk I gave at Tech2020, in Oak Ridge, Tennessee in June, 2012.

Text from the Slides:

Money, Process, and Culture
June, 2012
Adrian Carr
• I’m a Software Developer with a passion for efficiency.

• Experience as a business analyst, project manager.

• Scrum Product Owner and Scrum Master

• Director of Software Development

• Founder and Organizer of Agile Knoxville (http://agileknoxville.com)

Why am I here?
• I’m not trying to sell you anything.
• I want to see local businesses be successful.
• I also want to see people enjoying their jobs and their lives.

2002….
• I’m working in Atlanta

2011…
• I’m working in Knoxville
• Offered the job of leading the development team at Voices Heard Media.
-Run it however you want…

I travel to L.A. again…
Step One:
• Start with more frequent releases (reduce batch size)
– Reduced cycle time
– Faster feedback
– Greater ability to respond to changes
– Increased efficiency
– Greater sense of urgency
– Defects don’t build up unnoticed

Step Two:
• Agile practices.
– Small cross-functional teams
– Frequent delivery
– Small batch sizes
– Daily standup meetings
– Demos and planning meetings every 2-4 weeks

The Agile Manifesto
• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.
Principles behind the Agile Manifesto
• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
• The most efficient and effective method of
conveying information to and within a development team is face-to-face conversation.
• Working software is the primary measure of progress.

• Continuous attention to technical excellence
and good design enhances agility.
• Simplicity–the art of maximizing the amount
of work not done–is essential.
• The best architectures, requirements, and designs emerge from self-organizing teams.

• At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts its behavior accordingly.

http://agilemanifesto.org/
Warning:
• Partially agile isn’t very agile.
• Agile practices without the principles will only give marginal benefits.
• A few lean concepts can make an amazing difference.
• After a lot of study on this, I’ve come to believe that this won’t happen without systemic cultural changes.

Lean Thinking
• The core idea is to maximize value while minimizing waste. Simply, lean means creating more value with fewer resources.

http://lean.org

• Eliminating waste creates processes that need less human effort, less space, less capital, and less time to make products and services at far less costs and with much fewer defects, compared with traditional business systems.

http://lean.org

Lean Cultural Changes
• Optimize the Whole

Lean Cultural Changes
• Culture of problem solving
– Problems shouldn’t carry blame, or they will be suppressed, and you can’t solve them.
– Stop going around them and solve them instead
• The fact that we know more today, and are more capable today, is good news about today, not bad news about yesterday. Ron Jeffries

• A manager at an auto plant that used to be a manager in a Toyota plant: “When I was a manager at the Toyota plant, I had 2000 problem solvers working in “the plant. Now I have 10. How am I going to compete?” From “Leading Lean Development”

Lean Cultural Changes
• Continuous improvement

• Who is responsible for quality at your company?

• Expect employees to look for waste and ways to eliminate it.

Lean Cultural Changes
• “Stop the line” culture

• This can be painful at first, but yields great results.

Lean Cultural Changes
• 5 Whys
– Powerful tool. To get to root cause.
– Avoid trap of turning into 5 Blames

• Try to keep it simple. Don’t try to solve a massive, multi-faceted problem with this.

Be Prepared
• This will probably be hard, and will probably cause some clashes.

• It will expose things that some people (including you) hold dear.

• This is where your company culture gets really important. Some people won’t fit this culture. Be compassionate.

Corporate Values
• Integrity

• Communication

• Respect

• Excellence

How do you create a great culture?
• Reward the values that matter.
• Eliminate waste and make your employee’s jobs easier
• Encourage employees to ask why and look for better ways of doing things.
• Care about people.
• Be a coach, not a policeman

A Few Resources
• Leading Lean Development By Mary and Tom Poppendieck

• The Lean Startup By Eric Reis

• http://lean.org
• http://agileknoxville.com

Thoughts on Agile Waterfall

Not long ago, a friend of mine sent me an article about Agile Waterfall.

I thought about this quite a bit, and here are my thoughts on it:

My view is that agile is not the only way, and that you should do what works for your company and situation, with the caveat that you create a culture that has these attributes:

  • Continuous improvement, where every single person is on board with getting better as an individual, a team, and as a company. If people aren’t on board with that, then they are probably on the wrong bus.
  • Look to eliminate waste. Waste is those things which add cost, but do not add appropriate value. This could be processes, unneeded documentation, hand-offs from one group to another when they should be working together, bugs that cause rework, waiting too long for feedback, company rules, or anything. Like at Toyota, it should be everyone’s job to find these things.
  • Professionalism and trust. Teams should earn trust and be given it.

These are basic tenets of Lean thinking. Now, if you do these things, you will be agile, eventually, because it is impossible to have a culture of continuous improvement and do pure waterfall software development for long. In fact, I will go so far as to say that if you do waterfall development, you are probably robbing your company, because you are wasting your company’s money. I say probably because I do not know your situation, but I cannot see a case where pure waterfall development makes economic sense.

I’ve thought about this waaaay more than I should, but that’s just what I do, and I’ve come to the conclusion that the next time I am involved in helping a company move to agile development, I’m rarely going to use the word “agile”. It sets some people off, and also comes with preconceived notions, both good and bad. There are so many different flavors and ways of being agile that they come with assumptions. My plan is to incorporate lean concepts, and do those things that I listed above. Most people have heard of lean, and almost no one will fight it because it has won. Companies that are seriously doing continuous improvement, will get to agile, then go beyond that into lean. The problem with agile is that too many companies implement Scrum, and think they are agile, when they are really just doing the practices, but still aren’t making the real improvements that they could.

Abrams Falls

Abrams Falls

Adding multiple files to Source Code Control from IntelliJ IDEA

I recently started a new software project using IntelliJ IDEA, and after some experimentation, decided to add the project to Team Foundation Server (TFS) so I can make changes and have them under version control. I had no problem setting up the connection to our TFS Server via the “VCS” menu, and I could easily add and checkin a directory or a single file by right-clicking on it in the project explorer window, but could not figure out how to add all of my project files and directories at the same time. Adding them one by one to version control is tedious, time-consuming, and error prone. Plus, I knew there was a way. I just didn’t know what it was.

After a lot of wasted time trying, and then googling, then checking Stack Overflow and the JetBrains support site, I eventually gave up and contacted their support.

Five minutes later, I received the answer back from them. (In addition to spending too much time fighting Eclipse, that’s one more reason why I like them so much.)
Here’s the answer:

  1. Look at the very bottom of the IDE, and see if there is a tab labeled “Changes”. If not, go to View > Tool Windows > Changes.
  2. Look at the “Local” panel in the “Changes” window.
  3. Under the “Unversioned files” node, there will be a list of all files and directories that are not under version control.
  4. Right-click the “Unversioned Files” node, and select “Add to VCS” from the pop-up menu.
  5. Right-click again, and select “Commit Changes”. You should be done.

I hope this helps someone, and remember- Version Control is your friend.
Adrian Carr

PicasaView instantView Google Error

I noticed a great photo thumbnail layout on a friend’s blog recently, and asked him about it. He told me that he was using a cool WordPress photo plugin called PicasaView that easily pulls photos from Picasa into a WordPress blog. I’ve been playing around with different photo galleries lately as I try to resurrect this blog, and thought I’d give it a try since wordpress plugins are so easy to install.

So, I went into my wordpress control panel and added, activated, and setup the plugin in just a few minutes. Easy as pie.

I then went to a test page and added a Picasa photo album from a recent trip to Phoenix, Arizona.
I added this code:
[picasaview album='Arizona-August 2011']
This works great, and looks just like this:

Arizona-August 2011

Arizona-August 2011

(07/30/2011, 42 Photos)

Now, I’m going to come back to it, but notice that that there is a space in my album name. That’s perfectly fine to both Google and to PicasaView, up to a point.
What I really wanted was a bunch of thumbnails like I had seen on my friend’s blog. So, I added the “instantView” option to my PicasaView tag so it looked like this:

[picasaview album='Arizona-August 2011' instantView='yes']
This is when things went downhill fast.
In addition to messing up the entire CSS for my site, including messing up the header and background, it also generated this:

PicasaView Error

PicasaView Error

The issue here took me some time to find. The problem is that I used the title of the album as I entered it. This is the part that doesn’t work, because I have a space in the album name. The fix is to go to the url for the album that Picasa generated, and use that instead in the album name. In this image, you can see the album name and the url that works:
picasa url

After I changed the PicasaViewer tag to look like this:
[picasaview album='ArizonaAugust2011' instantView='yes']
it generates thumbnails like this:

Arizona-August 2011

Viewing images 1-42 of 42
IMG_20110807_165753.jpg IMG_20110807_191845.jpg IMG_20110807_192236.jpg IMG_20110807_192252.jpg IMG_20110807_192539.jpg IMG_20110807_192732.jpg IMG_20110807_192925.jpg IMG_20110807_193328.jpg IMG_20110807_193725.jpg IMG_20110807_193753.jpg IMG_20110807_194759.jpg IMG_20110807_195731.jpg IMG_20110807_202617.jpg IMG_20110808_190529.jpg IMG_20110808_190543.jpg IMG_20110808_190734.jpg IMG_20110808_191049.jpg IMG_20110808_191057.jpg IMG_20110808_191124.jpg IMG_20110808_191133.jpg IMG_20110808_191532.jpg IMG_20110808_191830.jpg IMG_20110808_192116.jpg IMG_20110808_192309.jpg IMG_20110808_192313.jpg IMG_20110808_192324.jpg IMG_20110808_193447.jpg IMG_20110808_193632.jpg IMG_20110808_194343.jpg IMG_20110808_194422.jpg IMG_20110808_194434.jpg IMG_20110808_194502.jpg IMG_20110808_194518.jpg IMG_20110808_194644.jpg IMG_20110808_194713.jpg IMG_20110808_202034.jpg IMG_20110808_202148.jpg IMG_20110809_193248.jpg IMG_20110810_130848.jpg IMG_20110811_152115.jpg IMG_20110811_172524.jpg IMG_20110811_180403.jpg
Viewing images 1-42 of 42

That’s all there is to it. I hope this helps someone else.
-Adrian

Better design and faster innovation

This statement caught my eye today. I look forward to the day where I am instrumental in making this happen. It sums up a large segment of my view of work.

We still went ahead because we knew we could design something that would simply be better and we could innovate light years faster. Now, we are rarely compared to this giant, and when we are, it’s because someone is saying we’re doing a better job. Our execution is so different that it, in and of itself, is reason enough for people to turn to us.
http://www.fastcompany.com/9-nagging-questions-to-tune-out-when-launching-a-startup

Image just for fun….
Superheros