How to Install and Run Elasticsearch as a Windows Service

Planning to keep this as simple and helpful as I can.
This whole process can be done in just a few minutes, assuming your downloads happen quickly.

Prerequisites:

  1. Log in to your server with credentials that allow software installation.
  2. Download Elasticsearch (latest. 1.5.2 used in this example) as a ZIP file from https://www.elastic.co/downloads/elasticsearch
  3. Unzip the file you just downloaded.
  4. Copy the child Elasticsearch1.5.2 folder and contents to C:\elasticsearch-1.5.2, or wherever you want it.
    • (You should have this path: D:\elasticsearch-1.5.2\bin )
  5. Download and install Java JDK 64 bit version from: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
  6. Set JAVA_HOME environment variable:
  7. java_home

  8. Install Elasticsearch as a service:
  9. (Here’s the original reference if needed)

    1. Run command prompt as administrator
    2. Navigate to C:\elasticsearch-1.5.2\bin
    3. Type service.bat install and hit Enter.
    4. It should install as a 64 bit service:
    5. es_console

    6. Close command prompt
    7. Open services console, and find Elasticsearch service.
    8. Open Elasticsearch service properties and set the service to start automatically and then start service.
    9. In a browser on the server, go to http://localhost:9200/
    10. You should see something like this (Note that the “name” is randomly selected from a list of around 3000 Marvel characters.):
    11. es_screenprint

  10. If you see that, you have successfully installed Elasticsearch in a cluster with one node, and you rock.
  11. Setting up multiple nodes is beyond the scope of this article.

    Elasticsearch is amazing, and running as a Windows Service is nice, because you don’t have to make sure you start it manually each time.

Amazing Speed: Elasticsearch for the .NET Developer. Codestock, 2015

This presentation is from a talk I gave at Codestock, 2015, which had over 900 attendees. There were between 50 and 60 people who attended this talk.

Here’s the text of the slides:
Amazing Speed:
Elasticsearch for the .NET Developer
4:05 PM / 301-D
Adrian Carr
With special guest- Shaunak Kashyap, Developer Advocate, Elastic.co
A little about me…
• Software Engineer & Team Lead
o The White Stone Group in Knoxville, TN.
• Previously
o Jewelry Television
o Alltel/Fidelity
o One terrible non-profit in Boone, NC that I don’t like to talk about.
• Several small startups

Experience:
• Full stack developer, C#, Ruby/Rails, JavaScript, Java, etc……. “Business Problem Solver”

A little background…
• We have some large clients, with lots of patients.

Really slow sometimes…
• Some legacy database decisions that weren’t so good back in the day.
• Worked just fine when volume was lower.
• Re-architecting would mean changing too many apps.
• Maybe you just have too much data to search efficiently, or it isn’t structured for search.

I’ve seen this before…

• Jewelry Television
• At the time, a lot of the database design wasn’t very good, but it worked.

o ~$200 million when I started in 2005
o ~$550 million three years later

The JTV Solution:

• It was awesome!

I’m a Business Problem Solver…

So….

A little research…
I found Elasticsearch -built on top of Lucene.

I found:
Wikipedia

And here:
GitHub

And here:
StackOverflow

What is Elasticsearch?
Elasticsearch is a distributed, open source search and analytics engine, designed for horizontal scalability, reliability, and easy management

So, I did a proof of concept

Terminology
• Node: DB Instance. (Java process running Elasticsearch)
• Cluster: Database Cluster- One or more nodes with same cluster name
• Index: Database, logical grouping of tables
• Type: Database Table
• Document: Like a row. JSON, key-value pairs
• Fields: Columns
• Shard: Worker processes. (These mostly happen automagically by Elastic)

Tools
• Elasticsearch (Obviously)
o Demo of how to install and run. (It’s easy.)

• Browser http://localhost:9200/, etc.

• Sense- a fantastic Chrome plugin

• C# with Nest

How to Load & Synchronize Data?
• It depends…

This is what I did:
• SQL Server Triggers on data that matters
• Windows Service
o Nest, C#, Elastic’s Bulk API (Found 10,000 rows at a time to be the sweet spot for insertion speed)

Gotchas
• Elasticsearch.net vs Nest
• Documentation on Elasticsearch is fantastic. Documentation on Nest is very sparse.

More Gotchas
• It’s a different way of thinking.
o Not RDBMS
o Joins? Nope
• Case sensitivity
• Index size- disk space can grow a lot
• Integration with existing application. Are your existing users going to have their cheese moved?

There is much more to know…

• Analyzers- These can be very complex
• Scaling- Elastic has a lot built in, but it can be tuned quite a bit.
• Field weighting- Relevance of blog post vs body vs comments
• This is just a start.

Questions?
• Code samples will eventually be at http://adriancarr.com
• Contact:

o adriancarr@gmail.com

o shaunak@elastic.co

o https://discuss.elastic.co/

Thank You!

Things your professors probably never told you, but will make you more money and help you have fun doing it.

AITP Presentation at Pellissippi State Community College- November, 2014

Great fun presenting this to around 50-60 students and faculty at PSCC. There was a good bit of commentary and discussion that’s not captured here, but this captures most of it.

Make it searchable:

Association of Information Technology Professionals
Pellissippi State Community College
November 13, 2014 Adrian Carr
Congratulations! This can be a fantastic career.
It can really suck at times too.
A lot of this is up to you:
Things your professors probably never told you, but will make you more money and help you have fun doing it.
A bit about me…
• I’m a software engineer, team lead….
– I’ve done other things…
• I also do some software consulting
• I started at a Community College- (Attended three of them)
It was hard, but worth it

I love my job
• Not every day, but almost every day.
Getting started
1.Please graduate. It opens doors, and I’ve met a lot of people who wish they had.
2.Learn skills.
3.Build your “brand”.

Create a great resume

A bad resume

Resume Tips
• 1-2 pages
• Make it clean and easy to read.
• Make it relevant – Tailor it for the position.
• No speling erors!
I’m gonna Google you…
These can help build your brand, or work against it.
• Facebook
• Twitter
• Personal blog
• Dr Brown’s student pages
• LinkedIn.com

Who is responsible for your career? • You are.
• IT folks == Professional problem solvers
• I learn something almost every day, and I usually have to teach it to myself.
Most technologies that I work with today didn’t exist when I started…
• C#
• JSON/XML
• jQuery/Angular
• Android
• Ruby/Rails
• HTML5/CSS3
• Bootstrap
• …..
Be purposeful about it
• Home projects
• Volunteer- inside and outside of work
• Consulting- This is a great way to learn new technologies. Create a website or mobile app for a small fee for your cousin’s auto body shop…
• Change jobs every few years.
• Do things outside your comfort zone
Even if you are on the right track, you’ll get run over if you just sit there. – Will Rogers
Be valuable
• Why are you being paid?
• Are you delivering value or filling a seat?
• If it was your money you were spending on your salary, would you change the way you work?
Most programmers have too many layers of abstraction between their code and their paycheck. – Michael Neel
Solve Problems
• Ask why.
• Figure out what problem people are trying to solve.
• Always look for better ways.
We cannot solve our problems with the same thinking we used when we created them. -Albert Einstein
Value
• Make your boss look good.
• Be worth so much more than you are paid that it’s no question about whether they should keep you or not.

Luck Surface Area:
Luck Surface Area is directly proportional to the degree to which you do something you’re passionate about combined with the total number of people to whom this is effectively communicated.
3. Build your “brand”.
“Soft” Skills
• Communication
• Attitude – “Whether you think you can or think you can’t, you’re right.“ -Henry Ford
• Humility goes a long way.

Be the type of person that people want to work with.
• Bring donuts or leftover cookies
• Be patient.

People may not remember exactly what you did or what you said, but they will always remember how you made them feel.
–Tony Hsieh, Zappos.com
Get involved
• AITP is a great start.
• Codestock
• Knoxville JS
• Agile Knoxville
• Write about things you learn- somewhere.
• StackOverflow.com
• Mentoring- High School, PSCC students,
http://techco.org/

A lot of this is up to you:
Things your professors probably never told you, but will make you more money and help you have fun doing it.
You can have options
If you do things right, you have options
• You can change where you work, or you can change where you work .
•”You have brains in your head. You have feet in your shoes. You can steer yourself, any direction you choose.” – Dr. Seuss
If you do really do things right…
• You can help set the course, and make the way we work better.
http://www.virgin.com/richard-branson/the-way-you-work-is-going-to-change
Questions?
Resources Job search/market research:
• http://indeed.com
Reading:
• Code Complete – Steve McConnell
• https://gettingreal.37signals.com/
• http://theleanstartup.com/
• Leading Lean Software Development – Mary and Tom Poppendieck
Contact: http://adriancarr.com

SQL Server- Check if Current Time is in Standard Work Week

Had to figure this out today, and thought I’d post it here, because I didn’t see it anywhere else out there.
There maybe better ways of doing this. If so, please let me know.

--Check to see if current time is a weekday and between 9 and 5
DECLARE @isWorkHours AS bit

SELECT @isWorkHours = COUNT(1)
WHERE
--Monday is 2 and Friday is 6.
DATEPART(dw,GETDATE()) BETWEEN 2 AND 6
AND
--This gets times between 9:00 and 4:59
DATEPART(hour, GETDATE()) BETWEEN 9 AND 16
IF @isWorkHours = 1
BEGIN
SELECT 1
END

IF @isWorkHours = 0
BEGIN
SELECT 0
END

Money, Process, and Culture- Lean and Agile Applied

This is the Powerpoint for a talk I gave at Tech2020, in Oak Ridge, Tennessee in June, 2012.

Text from the Slides:

Money, Process, and Culture
June, 2012
Adrian Carr
• I’m a Software Developer with a passion for efficiency.

• Experience as a business analyst, project manager.

• Scrum Product Owner and Scrum Master

• Director of Software Development

• Founder and Organizer of Agile Knoxville (http://agileknoxville.com)

Why am I here?
• I’m not trying to sell you anything.
• I want to see local businesses be successful.
• I also want to see people enjoying their jobs and their lives.

2002….
• I’m working in Atlanta

2011…
• I’m working in Knoxville
• Offered the job of leading the development team at Voices Heard Media.
-Run it however you want…

I travel to L.A. again…
Step One:
• Start with more frequent releases (reduce batch size)
– Reduced cycle time
– Faster feedback
– Greater ability to respond to changes
– Increased efficiency
– Greater sense of urgency
– Defects don’t build up unnoticed

Step Two:
• Agile practices.
– Small cross-functional teams
– Frequent delivery
– Small batch sizes
– Daily standup meetings
– Demos and planning meetings every 2-4 weeks

The Agile Manifesto
• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.
Principles behind the Agile Manifesto
• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
• The most efficient and effective method of
conveying information to and within a development team is face-to-face conversation.
• Working software is the primary measure of progress.

• Continuous attention to technical excellence
and good design enhances agility.
• Simplicity–the art of maximizing the amount
of work not done–is essential.
• The best architectures, requirements, and designs emerge from self-organizing teams.

• At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts its behavior accordingly.

http://agilemanifesto.org/
Warning:
• Partially agile isn’t very agile.
• Agile practices without the principles will only give marginal benefits.
• A few lean concepts can make an amazing difference.
• After a lot of study on this, I’ve come to believe that this won’t happen without systemic cultural changes.

Lean Thinking
• The core idea is to maximize value while minimizing waste. Simply, lean means creating more value with fewer resources.

http://lean.org

• Eliminating waste creates processes that need less human effort, less space, less capital, and less time to make products and services at far less costs and with much fewer defects, compared with traditional business systems.

http://lean.org

Lean Cultural Changes
• Optimize the Whole

Lean Cultural Changes
• Culture of problem solving
– Problems shouldn’t carry blame, or they will be suppressed, and you can’t solve them.
– Stop going around them and solve them instead
• The fact that we know more today, and are more capable today, is good news about today, not bad news about yesterday. Ron Jeffries

• A manager at an auto plant that used to be a manager in a Toyota plant: “When I was a manager at the Toyota plant, I had 2000 problem solvers working in “the plant. Now I have 10. How am I going to compete?” From “Leading Lean Development”

Lean Cultural Changes
• Continuous improvement

• Who is responsible for quality at your company?

• Expect employees to look for waste and ways to eliminate it.

Lean Cultural Changes
• “Stop the line” culture

• This can be painful at first, but yields great results.

Lean Cultural Changes
• 5 Whys
– Powerful tool. To get to root cause.
– Avoid trap of turning into 5 Blames

• Try to keep it simple. Don’t try to solve a massive, multi-faceted problem with this.

Be Prepared
• This will probably be hard, and will probably cause some clashes.

• It will expose things that some people (including you) hold dear.

• This is where your company culture gets really important. Some people won’t fit this culture. Be compassionate.

Corporate Values
• Integrity

• Communication

• Respect

• Excellence

How do you create a great culture?
• Reward the values that matter.
• Eliminate waste and make your employee’s jobs easier
• Encourage employees to ask why and look for better ways of doing things.
• Care about people.
• Be a coach, not a policeman

A Few Resources
• Leading Lean Development By Mary and Tom Poppendieck

• The Lean Startup By Eric Reis

• http://lean.org
• http://agileknoxville.com

Cape Point Campground- Cape Hatteras National Seashore- OBX

In July, 2012, we traveled to the Outer Banks of North Carolina. We camped one night at Oregon Inlet Campground at Cape Hatteras National Seashore. We liked that campground quite a bit, so one day we went to check out two other US Park Service campgrounds, Frisco, and Cape Point.

All three campgrounds are very different. Cape Point is one giant field. This campground resembles a giant football field, with paved driveways and paved parking areas for campsites running through it. There are bathhouses and showers scattered around every once in a while. There are no trees whatsoever, and absolutely no shade, unless you bring an awning of some sort. It could be very exposed in the event of a storm or high winds.

The grass was in really good shape, and would be great for frisbee or kicking a soccer ball around. I don’t know if there are a lot of sandspurs like there are at Oregon Inlet.

We rode through on Thursday, July 12, and very, very few sites were taken.

Cape Point is at the corner of Hatteras Island, south of Oregon Inlet and north of Cape Point. The famous Hatteras lighthouse is just down the road, and very visible from the camground.
Here is a good map of Hatteras National Seashore.

Cape Point Campground- Hatteras National Seashore -OBX

Viewing images 1-6 of 6
Cape Point Campground1.jpg Cape Point Campground2.jpg Cape point Campground Entrance.jpg Hatteras Beach.jpg Hatteras Beach3.jpg Hatteras Lighthouse.jpg
Viewing images 1-6 of 6

Frisco Campground- Cape Hatteras National Seashore, OBX

In July, 2012, we traveled to the Outer Banks of North Carolina. We camped one night at Oregon Inlet Campground at Cape Hatteras National Seashore. We liked that campground quite a bit, so one day we went to check out two other US Park Service campgrounds, Frisco, and Cape Point.

All three campgrounds are very different. Frisco is the only one with trees, and has some sites that are a bit higher, and have a view of the ocean across the campground. The downside is that these sites are a bit further from the ocean, but still an easy walk.

Frisco Campground has some beautiful campsites where a tent could actually be completely in the shade, at least at many of the sites. We rode through on Thursday, July 12, and very few sites were taken. We could have had our pick of excellent sites.

Frisco is at the tip of Hatteras Island, south of Oregon Inlet and Cape Point.
Here is a good map of Hatteras National Seashore.

We also saw a deer at a campsite.

Frisco Campground- Hatteras National Seashore- OBX

Viewing images 1-5 of 5
DSC01333.JPG Frisco Campground.jpg Frisco Campground2.jpg Frisco Campground3.jpg Frisco Campground4.jpg
Viewing images 1-5 of 5

Camping at Oregon Inlet- Cape Hatteras National Seashore, OBX

In July 2012, I traveled to the Outer Banks of North Carolina. It was kindof a last minute trip, and we had no reservations lined up at all. We thought we might find a small motel for the first night, but passed by the Oregon Inlet Campground and decided to check it out. Campsites at Oregon Inlet can’t be reserved ahead of time (except for group sites) and before the trip, I tried to find out the likelihood of finding an open spot during high season on the Outer Banks but couldn’t get a definitive answer. I tried calling the park service, and eventually navigated the phone tree through, but was prompted to leave a message, which I declined to do. I even posted a question to the OBX forum on TripAdvisor, but got no answer. Anyway, it turned out that there were plenty of open campsites available, and the campground was quite nice.

We arrived at around 5:30 on a Saturday afternoon in July, and found that less than 1/4 of the spots were taken. We rode through, found a spot that was at least somewhat sheltered from the wind by some large dunes, and decided to stay the night for $20 instead of $150 at a dumpy motel. Setting up the tent was a challenge, because the tent is quite large, and it was quite windy, and very hot. Fortunately, I had two great helpers to help me set it up. We then went to dinner in the town of Salvo, which is about 15-20 minutes away. We found a decent restaurant with a second story where we could watch the kiteboarders in the sound.

When we got back to the campground, more people had arrived, and probably a third of the campsites were taken. It was quite dark by this time, so we went to sleep. The wind was still very strong, and whipped the tent constantly all night, making it difficult to sleep due to the noise. The breeze did give some respite from the heat. If it wasn’t for the breeze, it would have been very hot and quite uncomfortable for sleeping.

In the morning, we walked across the dunes to the beach. It was beautiful. One thing that is a bit strange is that there is no designated trail to the beach, so you just have to find one and follow it. However, to get to the trails, you usually have to walk through or very close to someone else’s campsite. The trails are beautiful themselves, as small winding paths through the dunes. It’s very different from walking down a traditional wooden boardwalk over the dunes. On the trails, you never know what is around the next bend, and they dip down and up and through the dunes.

Once through the dunes, we were greeted by an empty beach, with only three other people on it as far as we could see. Not bad for a Sunday morning in July.

Be sure to use serious tent stakes. We saw more than one tent that had blown over on their side, and one of them looked like it had blown across the site and come to rest against the dunes. I didn’t have large tent stakes, so I found some sticks and filled a couple empty water bottles with sand, and tied the guy lines to these and buried them about a foot deep in the sand. Fortunately, I had brought a small garden shovel for digging at the beach.

About Oregon Inlet campground:

  • Near Oregon Inlet marina and fishing station.
  • Cost is $20 per night
  • The NPS website says that mosquitos can be bad, but weren’t while we were there.
  • Cold showers.
  • Decent, clean bathrooms. (No soap or hand towels)
  • Walk through the dunes to get to beach. May have to walk through someone else’s site to get to trail.
  • Many sandspurs, so be careful.
  • Have to use large tent stakes, or use tent guy lines tied to a stick and buried in the sand.
  • No trees.
  • Some grassy areas, and some mostly sand.

Oregon Inlet Campground- Hatteras National Seashore- OBX

Viewing images 1-18 of 18
DSC01162.JPG DSC01163.JPG DSC01168.JPG DSC01169.JPG DSC01170.JPG DSC01172.JPG IMG_20120708_071417.jpg IMG_20120708_071422.jpg IMG_20120708_071428.jpg IMG_20120708_071513.jpg IMG_20120708_071544.jpg IMG_20120708_071845.jpg IMG_20120708_072033.jpg IMG_20120708_072214.jpg IMG_20120708_073008.jpg IMG_20120708_073013.jpg Oregon Inlet Campground Entrance.jpg IMG_20120708_071323.jpg
Viewing images 1-18 of 18

Thoughts on Agile Waterfall

Not long ago, a friend of mine sent me an article about Agile Waterfall.

I thought about this quite a bit, and here are my thoughts on it:

My view is that agile is not the only way, and that you should do what works for your company and situation, with the caveat that you create a culture that has these attributes:

  • Continuous improvement, where every single person is on board with getting better as an individual, a team, and as a company. If people aren’t on board with that, then they are probably on the wrong bus.
  • Look to eliminate waste. Waste is those things which add cost, but do not add appropriate value. This could be processes, unneeded documentation, hand-offs from one group to another when they should be working together, bugs that cause rework, waiting too long for feedback, company rules, or anything. Like at Toyota, it should be everyone’s job to find these things.
  • Professionalism and trust. Teams should earn trust and be given it.

These are basic tenets of Lean thinking. Now, if you do these things, you will be agile, eventually, because it is impossible to have a culture of continuous improvement and do pure waterfall software development for long. In fact, I will go so far as to say that if you do waterfall development, you are probably robbing your company, because you are wasting your company’s money. I say probably because I do not know your situation, but I cannot see a case where pure waterfall development makes economic sense.

I’ve thought about this waaaay more than I should, but that’s just what I do, and I’ve come to the conclusion that the next time I am involved in helping a company move to agile development, I’m rarely going to use the word “agile”. It sets some people off, and also comes with preconceived notions, both good and bad. There are so many different flavors and ways of being agile that they come with assumptions. My plan is to incorporate lean concepts, and do those things that I listed above. Most people have heard of lean, and almost no one will fight it because it has won. Companies that are seriously doing continuous improvement, will get to agile, then go beyond that into lean. The problem with agile is that too many companies implement Scrum, and think they are agile, when they are really just doing the practices, but still aren’t making the real improvements that they could.

Abrams Falls

Abrams Falls

Adding multiple files to Source Code Control from IntelliJ IDEA

I recently started a new software project using IntelliJ IDEA, and after some experimentation, decided to add the project to Team Foundation Server (TFS) so I can make changes and have them under version control. I had no problem setting up the connection to our TFS Server via the “VCS” menu, and I could easily add and checkin a directory or a single file by right-clicking on it in the project explorer window, but could not figure out how to add all of my project files and directories at the same time. Adding them one by one to version control is tedious, time-consuming, and error prone. Plus, I knew there was a way. I just didn’t know what it was.

After a lot of wasted time trying, and then googling, then checking Stack Overflow and the JetBrains support site, I eventually gave up and contacted their support.

Five minutes later, I received the answer back from them. (In addition to spending too much time fighting Eclipse, that’s one more reason why I like them so much.)
Here’s the answer:

  1. Look at the very bottom of the IDE, and see if there is a tab labeled “Changes”. If not, go to View > Tool Windows > Changes.
  2. Look at the “Local” panel in the “Changes” window.
  3. Under the “Unversioned files” node, there will be a list of all files and directories that are not under version control.
  4. Right-click the “Unversioned Files” node, and select “Add to VCS” from the pop-up menu.
  5. Right-click again, and select “Commit Changes”. You should be done.

I hope this helps someone, and remember- Version Control is your friend.
Adrian Carr